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Preface 



Section 106 of the Job Training Partnership Act (JTPA) directs the 
Secretary of Labor to establish performance standards for adult, 
youth, and dislocated worker programs. In accordance with the Act, 
the secretary of Labor established national performance standards 
based on Comprehensive Employment and Training Act (CETA) data for 
the initial nine-month period of JTPA and for the first program 
year. For Program Year 1986 (PY86) , the national standards for 
Title II A (and a goal for Title III) are based on PY84 JTPA data. 

These data reflect actual experience of SDAs. Now that JTPA has 
a performance history, performance standards have become in- 
creasingly important to program administrators and policy makers 
as management tools and benchmarks of outcomes. 

No issue has received more attention from States and Service 
Delivery Areas (SDAs) than this one. The policy issues involved 
in developing and implementing a performance standards system 
have been the subject of much deliberation and debate. 

In an attempt to meet the legislative intent of JTPA to increase 
employment and earnings of participants and decrease dependency 
on welfare programs, the National Association of Counties (NACo,, 
the National Governors' Association (NGA) and the National 
Association of Private Industry Councils (NAPIC) have joined 
together to develop and disseminate guidance to help state and 
local level managers in the JTPA delivery system, to venture 
"beyond the model." Practitioners have found that in some in- 
stances, further adjustments are needed beyond the Department of 
Labor's adjustment methodology. 

This Technical Assistance Guide was designed to provide assistance 
in planning and developing negotiated JTPA performance standards. 
The performance standards system and the DOL adjustment model 
attempt to accomodate service to the more disadvantaged, 
(particularly individuals with less education and other hard to 
serve" groups), and more extensive and more comprehensive services, 
by holding SDAs harmless for the generally lower outcomes ex- 
perienced by such participants. It is possible for states, 
service delivery areas, and private industry councils, however, 
to make further adjustments and cooperate creatively in nego- 
tiating standards that will support and enhance both local and 
state goals. 

Performance standards are generalized benchmarks or measures, but 
they are prescriptive by nature. They are based on data that can 
provide information on relative and absolute program performance. 
They can provide a framework for program planning. They also pro- 
vide a measure cf progress. Performance standards adjustments 
can be a valuable tool to assist states and service delivery 
areas in planning programs and in serving the "hardest to serve.' 
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This Technical Assistance Guide was developed for planners and 
administrators at both the state and local service delivery area 
levels. It should be of particular interest to private industry 
council staff who might have concerns over service mix or taraet 
populations. ^ 

The idea for a Technical Assistance Guide that ventures "Beyond 
the Model and provides "An Approach to Negotiating Performance 
Standards emmated from conversations with members of the United 
States Department of Labor Advisory Committee on Performance 
Standards. Kay Albright, formerly Director of the Office of 
Performance Standards and Program Review of the U.S. Department 
of Labor and now Research Associate with the National Governors' 
Association, Lori Strumpf, Assistant Director of the National 
Association of Private Industry Councils, and Roxie Nicholson, 
formerly Senior Research Associate at NACo, now Policy Analyst at 
the U.S. Department of Labor agreed to devote time to develop 
this manual. 



Ms. Albright served as principal author of Chapter 1, introduction. 
Ms. Strumpf developed Chapter 2 on planning and performance stan- 
dards. Ms. Nicholson provided overall guidance and review of the 
technical content, along with all the authors. Jeff Zornitsky, 
Senior Economist, Abt Associates, performed the technical work 
and developed the core material for chapters three and four. 
This Guide represents a true joint venture, with all that that 
implies, among NACo, NGA and NAPIC. All three organizations 
contributed both time and staff resources to make this Guide 
possible; NACo provided the financial resources for the technical 
consultants, and for printing and distribution. 

The authors of this manual wish to thank the manv people at the 
national, state, and local level, particularly Sue Manzo, 
Nancy Bross and Chris King, who reviewed the content and provided 
their thoughtful suggestions. We particularly wish to thank 
Hugh Davies and Karen Greene of the U.S. Department of Labor for 
their outstanding work in performance standards and their input 
into this Guide. Despite the fact that we did not always agree 
on issues, the professional discussion greatly enhanced the final 
product. We also wish to thank Janet Reingold, j.r. Reingold 
and Associates, inc. for editing the final manuscript under 
stringent time pressures. All of the authors worked on this 
project m hopes that it provides benefits to the practitioners 
in the training and employment system. 

The usual disclaimers apply. While we received valuable input 
and guidance from dedicated individuals within the U.S. Department 
of Labor and elsewhere, responsibility for the content (and any 
errors herein) rests solely with the authors. This Guide does not 
represent the official policy of the Department~of~Labor or any 
of the sponsoring organizations . Furthermore, since this entire 
Guide ventures "Beyond the Model", states and SDAs are cautioned 
that individualized approaches may remove the protections that 
are afforded by the model. 



ERIC 



8 



Use of This Guide 



This Technical Assistance Guide was developed for use by State, 
SDA and PIC staff who play a role in the process of planning, 
administering programs, or monitoring performance against 
standards. 

It will most probably be read by both novices and experts in 
the field. For those who are new initiates to the field of 
performance standards, the authors recommend reading the entire 
Guide. For those practitioners who are seasoned experts in 
the field, the authors recommend reading only chapters 3 and 4. 

Regardless of the level of expertise of the reader, it is recom- 
mended that this Guide be used in conjunction with (not m place 
of) the Department of Labor's Guides for Setting JTPA Title 
II-A Performance Standards for PY 1985 and PY 1986. 

Users of this Guide are urged to keep in mind that the DOL PY85 
and PY86 adjustment models reflect actual experience under JTPA. 
However, it is too early to determine all of the impacts of the 
PY86 model, and the extent of the further adjustments that will 
be required. 

We are just beginning to develop experience using the PY86 
adjustment model. It is likely that practitioners will be able 
to offer solutions to some of the issues that we are unable to 
solve at this time. Your comments and input are welcomed. 
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The Job Training Partnership Act (JTPA) delivery system is 
constantly challenged to improve the efficiency and effective- 
ness with which it operates. Congress and the public hold 30b 
training program operators accountable for their investment in 
human capital, and the return on this investment. 

It is within this context that the Secretary of Labor is required 
to develop national program performance standards, and Governors 
are permitted to prescribe variations to adapt these standards 
as necessary to their own specific economic, geographic, and 
demographic circumstances. 

The purpose of this Guide is to provide state and local level 
prcgram administrators and planners with an approach to de- 
veloping meaningful and relevant negotiated performance stan- 
dards that will serve as a management and evaluation tool. 

The premise is that national performance standards are necessary 
and are useful as a guage to judge nationwide success of JTPA. 
However, local service delivery areas face factors and circum- 
stances unique to a specific jurisdiction which may need to 
be taken into consideration in planning, operating, and evalua- 
ting local programs. Thus, DOL developed the adjustment models 
which take into account some of the unique local circumstances. 

This Guide is designed to offer some technical assistance to 
program officials to venture beyond the models in setting local 
standards. The system permits adaptation to local needs; this 
flexibility provides enormous opportunity. 

This Guide was developed jointly by the National Association of 
Counties (NACo) , the National Governors' Association (NGA) , and 
the National Association of Private Industry Councils (NAP IC) 
in the belief that states, service delivery areas, and private 
industry councils may want to use the flexibility within the 
system to fine-tune their own program planning and management 
systems, and that the performance standards adjustment process 
provides a vehicle to do so. 

Chapter I provides some background information on the legisla- 
tive and"administrative history and evolution of performance 
standards. It provides an overview of the performance stan- 
dard system as set forth oy the U.S. Department of Labor, 
including the measures, and the procedural parameters. Finally, 
it discusses performance standards within the context of overall 
progrcim management. 

Chapt'jr II examines performance standards as an integral part of 
the program planning process. It reviews the functions of 
strategic and operational planning and the ways that program 
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officials can select their strategies to meet local needs and 
circumstances, rather than to meet nationally prescribed numeric 
goals. Performance is the ultimate focus, not txie performance 
standards. 

Chapter ill provides the framework for a negotiated approach to 
setting performance standards. It examines the Department of 
Labor's adjustment model, its major limitations (such as 
targeting, labor market conditions, and program intensity), and 
the key issues or groundrules for undertaking a negotiated 
approach. 

Chapter IV focuses on the negotiation process. A negotiated 
approach to setting JTPA performance standards can significantly 
benefit both states and SDAs. It can be used to support state 
and SDA targeting policies to serve those most in need of 
training and employment assistance. The chapter discusses several 
important issues that states and service delivery areas must 
resolve when negotiating their performance standards. These 
issues include: establishing the basis for negotiation; de- 
veloping tools for further quantifying adjustments to the model- 
predicted performance scores; negotiation and its relationship 
to six percent incentive funds; establishing a framework to 
support a negotiation process; and state and SDA roles and respon- 
sibilities in establishing a negotiation process. 
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CHAPTER I 
Overview of Performance Standards 



INTRODUCTION 



Section 106 of the Job Training Partnership Act (JTPA) requires 
the Secretary of Labor to select perfo3nnance raeasures and 
establish national standards for adult and youth training pro- 
grams under Title II-A and dislocated worker programs under 
Title III. On the basis of the Secretary's performance stan- 
dards. Governors must set standards for each of their service 
delivery areas (SDAs) . For each program year, the Secretary 
provides revisions to the Department's adjustment methodology 
for the standards, based upon the newly available JTPA data. 

JTPA provides that the Governor has the right and the respon- 
sibility to adjust the national performance standards to meet 
particular local economic, geographic, and demographic conditions. 

Chapter I of this Guide provides some background infoinnation for 
state and local program managers and planners, on the origins 
of the current performance standards system, and the applica- 
tion of these standards within the context of overall JTPA 
program management. 



BACKGROUND 



The idea of measuring the results of training and employment 
programs is not new. Since their inception in the early 1960's, 
training and employment programs have been the subject of in- 
tensive research and evaluation, all designed to determine the 
effectiveness of various approaches. It has also been common 
practice to set programmatic goals, particularly at the program 
operator level, as part of the planning process. 

During the 1970 's, under the Comprehensive Employment and Training 
Act (CETA) , the Department of Labor (DOL) made serious efforts 
to define program success and to determine how it should be 
measured at the local level. Several alternative methods of 
establishing realistic expectations for local programs were 
tried. 

The traditional approach to deteinnine programmatic success was 
to assess how well a program performed at the end of a year 
against its planned goals for a particular objective such as 
job placements. This approach, assessing "planned vs. actual" 
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performance, had some clear limitations. The primary limita- 
tion was that "planned vs. actual" assessments had little to do 
with whether the level (or number) of results or outcomes was 
appropriate given the client population. Instead, this approach 
simply analyzed whether the program had made good planning 
judgments. 

Another approach used to negotiate numerical goals was the in- 
cremental approach, which required incremental improvement in 
performance over the prior year's performance (usually expressed 
in terms of a certain percentage improvement) . The problem 
with this approach is that it is unfair to programs which are 
already performing well. It is much more difficult to improve 
by 10 percent if the program has a 75 percent entered employ- 
ment rate than to improve by 10 percent if the program has a 
40 percent entered employment rate. The incremental method 
does not work well for all programs, but it does have some 
value for programs which need to improve performance sub- 
stantially. 

As training and employment professionals recognized the limi- 
tations of past approaches, other methods of assessing performance 
were tested. Much of the impetus to improve how CETA programs 
were assessed resulted from the 1977 CETA amendments which 
required performance standards for the first time. Early 
attempts to move beyond "planned vs. actual" included the 
identification of factors believed to influence performance 
and negotiation between prime sponsors and DOL regional offices 
to change individual entered employment rates based on these 
factors. A CETA Performance Standards Workgroup was formed in 
1980 to attempt to develop a method to quantify the factors 
believed to affect performance. This effort was based on work 
done in the state of Massachusetts to use multiple regression 
analysis to identify and quantify factors affecting performance. 
This regression model was used as a benchmarking system by the 
DOL regional offices in 1980 to assess prime sponsor performance. 



LEGISLATIVE BASIS UNDER JTPA 



The emphasis on performance in the Act is clear. Section 106 
outlines in detail the time frame for development of performance 
standards, the content of the measurement system, and the respon- 
sibilities for a performance-based system. Section 106 of 
the Act characterizes the JTPA program as an investment, the 
return on which is to be measured by participants' gains in 
employment and earnings and reductions in welfare dependency. 
(See Appendix A, which excerpts Section 106 of JTPA.) 

The approach to performance standards under JTPA differs from 
earlier attempts to gauge performance in two principal ways. 
First, JTPA emphasizes performance, or program outcomes rather 
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than the processes used to achieve these outcomes. The intent 
of the Congress was to develop criteria for measuring the 
return on the investment in human capital. The process was 
only a means to achieve the desired result. 

Second, JTPA suggests a systemmatic , uniform approach with 
factors and measures that comprise the standards and prescribed 
methods for calculating the expected outcomes. Under JTPA, 
there is an objective tool (the DOL adjustment methodology) 
for determining how well a local administrative entity performed, 
given its local conditions. 



OVERVIEW OF THE PERFORMANCE STAMDARDS SYSTEM 



In accordance with the statutory directives, the Secretary of 
Labor established seven performance measures. In addition, DOL 
issued national numerical standards for each measure. The 
national standards for the initial nine months of JTPA and for 
the first two program years were developed from FYS 2 CETA data. 
For program year 1986, the Department revised the national 
niixnerical standards based on PY84 JTPA II-A performance data. 

The Secretary's performance measures and associated standards 
for PY86 are: 

Adults 

' 0 ETibered Employment Rate - Number of adults who 

entered employment as a per cent cge of the number 
of adults who terminated: 62 percent. (55 per- 
cent in PY85) 

0 Cost per Entered Employment - Total expenditures 
for adults divided by the number of adults who 
entered employment: $4,374. ($5,704 in PY85) 

0 Average Wage at Placement - Average wage for all 
adults who entered employment at the time of ter- 
mination: $4.91. (same in PY85) 

0 Welfare Entered Employment Rate - Number of adult 
welfare recipients who entered employment at termi- 
nation as a percentage of the number of adult 
welfare recipients who terminated: 51 percent 
(39 percent in PY85) 

Youth 

0 Entered Employment Rate - Number of youth v/ho 

entered employment at termination as a percentage 
of the number of youth who terminated: 43 percent. 
(41 percent in PY85) 



15 



• Positive Termination Rate - Number of youth who 
entered employment at termination, plus the 
number of youth who met one of the youth employ- 
ability enhancement definitions at termination: 
75 percent (82 percent in PY85) 

(Youth employability enhancements include: 

a) attained PIC recognized youth employment 
competencies 

b) entered non-Title II training 

c) returned to full-time school 

d) completed major level of education 

e) completed program objectives (14-15 
year olds) ) 

• Cost per Positive Termination - Total expenditures 
for youth divided by the number of youth who en- 
tered employment at termination plus the number of 
youth who met one of the youth employability 
enhancements definitions at termination (see 
above): $4,900. (same in PY85) 

The real "standards" are those established by the state for or 
with local service delivery areas (SDAs) . Performance against 
these standards determines whether the SDA is eligible to 
receive an incentive award or will require technical assistance. 
JTPA provides the Governor both the right and the responsibility 
to adjust the national performance standards to meet conditions 
in the particular service delivery area. The Governor may 
adjust the national standards within parameters established by 
the Secretary as specified in Section 106(e) based upon: 

"specific economic, geographic, and demographic fac- 
tors in the State and in service delivery areas with- 
in the State, the characteristics of the population 
to be served,, and the type of services to be provided." 

The parameters established by DOL include the following: 

1. Procedures must be: 

• Responsive to the intent of the Act, 

• Consistently applied among SDAs, 

• In conformance with widely accepted statis- 
tical criteria; 

2. Source data must be: 

• Of public use quality, 

• Available upon request; 
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3. Results must be: 

• Documented , 

• Reproducible ; and 

4. Adjustment factors must be limited to: 

• Economic factors / 

• Labor market con^ILtionS/ 

• Characteristics of the population to be served^ 

• Geographic factors , 

• Types of services to be provided. 

To assist states in establishing local perfoannance standards 
which take into account some of the economic and demographic 
factors believed to influence perf oannance, DOL provided the 
same type of regression models as developed under CETA. The 
models could be used by states at their option to make initial 
adjustments for local factors, or they could use the unadjusted 
national standards, or vjhatever other approach they choose. 

States which used the models were also allowed to make further 
adjustments for local conditions as long as they followed the 
procedures (the parameters) described earlier. This approach 
allowed the national level to establish specific measures used 
by all programs, but allowed maximvim discretion to the states 
to enable SDAs to design progrcims which would meet the needs of 
their clients locally • 

The perfoirmance standards system allows the state to make 
judgments about which programs were a "success," not by rank 
ordering SDAs on a specific measure but by deteannining how 
much the SDA actually varied from the model-predicted standard 
(plus any other Governor's adjustment). (For example, an SDA 
with a model-derived entered employment rate of 40 percent and 
an actual entered employment rate of 60 percent really performed 
better than an SDA with a model-derived entered employment rate 
of 51 percent and an actual entered employment rate of 60 
percent given the apparent differences in local conditions.) 

One of the hallmarks of JTPA is the strong emphasis on perfor- 
mance — not necessarily performance standards. Standards are 
primarily a management tool for state and local program opera- 
tors, and are one way for the Congress to know whether the 
investment in job training programs is worthwhile. Performance 
standards are flexible enough to allow Private Industry Councils 
and local program administrators to serve clients of their own 
choosing using interventions deemed appropriate to obtain 
employment in order to increase both the employment and earnings 
of the participants. 
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CHAPTER II 

Performance Standards and the Planning Process 



INTRODUCTION 

Chapter II examines the performance standards within a program 
planning context. This chapter explores the planning process 
with performance standards as one functional element within 
that process. The model-derived performance standard, or num- 
ber, should be viewed as an expectation based on local conditions. 
If local conditions warrant the use of other data or additional 
factors, the planning process should provide this opportunity. 



PLANNING OVERVIEW 

Planning means choosing from alternative courses of action in 
order to reach selected goals . Generally there are two types 
of planning: 

• Strategic planning ~ a long term process for defining 
overall missions, goals and objectives, and 

9 Operational planning ~ a shorter term process for 
managing programs on an on-going, day-to-day basis. 

Strategic planning functions may include: 

- defining mission 

- setting goals and objectives 

- analyzing the current labor market and trends 

- identifying available resources 

- identifying overall strategies to achieve goals 

Operational planning functions may include: 

- fine tuning program plans, including numbers 
to be served, program mix, duration and 
sequencing, and client mix 

- selecting service providers and negotiating 
contracts 

- training staff and delegating responsibilities 



ERIC 



18 



-9- 



PLANNING STEPS 



Generally, the process for planning is a continuum along which 
important decisions must be made. The process includes: 

• Needs As sessment ; The assessment may focus on target 
client groups, employers, the community, the labor 
market, or any combination of these. Using available 
information on the labor market experiences of the 
disadvantaged, as well as lessons from previous pro- 
gram years, states and SDAs identify the universe of 
need and the nature and causes of the hardships in the 
labor market. The needs assessment provides the 
rationale for the programmatic approach by setting 
the framework for program objectives. 

• Assessme nt of Resources : Once the needs are identified, 
the resources available to address these needs must be 
considered. This includes organizational (staff, 
financial) resources, community resources, and 
linkages. An assessment of this type will help define 
realistic parameters for setting program goals and 
objectives, and maximize coordination. 

• Goal Setting; Using the data collected on needs and 
resources, within the framework of the organizational 
mission, the formulation of realistic program goals 
is possible. Goals should be clearly stated, as they 
guide program operations and evaluation efforts. A 
number of important decisions must be made during 
the goal setting process: 1) who will be served; 

2) how will they be served; and 3) what are the 
expected outcomes. The decisions made about these 
three areas will help shape the program strategy. 

• Operations ; With goals and objectives identified, it 
is now possible to plan management systems, program 
elements (services, who will deliver them, the best 
mix of services for the particular target group), and 
specific tasks. The overall program strategy should 
reflect the outcomes of the preceding planning steps. 

• Evaluation ; Finally, the program results should be 
measurable. The required performance standards will 
set some indicators of program success, but a method 
for measuring success, including management or staff 
performance, must be considered. 

Where does the calculation of performance standards fit into the 
planning process? If the performance standards dictate the pro- 
gram decisions, the planning process may be constrained. This 
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may do a disservice to potential participants as well as to 
the employer community. Performance standards must follow 
decisions about who to serve, with what types of strategies. 
It must be noted that one of the major planning differences 
between CETA and JTPA is that under JTPA, planning is based on 
terminee (as opposed to participant) characteristics. Planning 
becomes a more sophisticated process because it is planning 
for outcomes as oppored to process. 

Exhibit A illustrates the steps in the planning process, and 
the use of performance standards as a functional element of 
planning and management. 

ESTABLISHING GOALS AND OBJECTIVES AND 
SELECTING PROGRAM STRATEGIES 



Upon assessing the potential labor market, identifying the 
eligible population, and assessing both client and employer 
needs, programs can set realistic goals and objectives that 
have the potential to match services to participants and to 
match participants to potential employers. It is at this point 
that the performance standards play an important, and initial 
role in the planning process. 

Program planners must assess the eligible population, and the 
labor market to determine who to serve and what services to 
provide. Then questions are answered within the overall pol.^cy 
framework that the PIC has established. 

For instance, program planners may be concerned about the large 
number of high school dropouts among both its youth and adult 
population. A board of education survey may document that the 
dropout rate is high. During the planning process, employers 
are surveyed to ascertain what they look for in entry level 
workers. The responses indicate that most employers want 
employees who have basic work habits, good attitudes and basic 
academic skills, particularly math and reading. An assessment 
of the JTPA eligible client pool indicates a higher proportion 
of dropouts in the eligible population than in the general 
population. Among the eligible dropouts two things are apparent- 
generally they read, write and compute at below grade level and 
they have poor work histories. 

Given this set of circumstances, one goal might be to target 
both adult and youth dropouts to increase basic skills and work 
experience so that they may get and keep entry level jobs in the 
community . 
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EXHIBIT A 

Performance Standards and the Planning Process 
TYPE OF PLANNING PLANNING STEPS 



Strategic 
Planning 



Performance^ 
Standards 



Operational 
Planning 



Performance 
Standards 



Define Mission 
Analyze Labor Market and 
Eligible Population 
Needs 



Establish Program 
Policy/Goals and 
Objectives 



Select Program 
Strategies (Mix, 



Duration/ Sequencing) 



I 



Develop Operational , 
Management and Evalua- 
tion Plans (Select 
Service Deliverers) 



Implement Programs/ 
Monitor Progress 



Evaluate Results 



21 



-12- 



Program strategies may include: 

• Basic skills remediation classes for youth 

• Adult basic education classes 

• Work experiences 

The anticipated program outcome for adults is placement and 
for youth in this instance is either attained youth competencies, 
return to school, or placement. However, the kind of training 
required may take longer than "average" and may cost more to 
achieve results. Rather than adjusting services provided, and 
target populations to fit the standard, program planners may 
adjust the standards to fit local needs. 

If the standard were imposed at the front end of the process, 
the planners might not exercise the flexibility available in 
choosing who to serve. 

The focus on performance under JTPA allows programs to develop 
a process by which they can manage their performance levels. 

Performance standards must be viewed within a local service 
delivery context. Under JTPA, program planners are in a position 
to identify the impact that they wish to have on the local 
eligible population and on the labor market. Performance 
standards can be viewed as a tool, not a "rule." Local 
decision making based on the needs of the eligible population 
and the needs of local employers can be accounted for within 
the existing performance standards system. And, the performance 
standards system can be molded to meet local needs and 
priorities. 



PROGRAM EVALUATION 



Performance standards also play a critical role during the 
evaluation phase of the planning and management cycle. They 
provide indicators of success, :nd some information on areas 
requiring further attention. This information should be fed 
back into the planning process. 
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CHAPTER III 

Beyond the Model: The Need for a Negotiated Approach to 
Setting Performance Standards 



INTRODUCTION 



program planning in the JTPA system has become more sophisti- 
cated and rigorous. Increased private sector involvement and 
growing cost consciousness among administrators and the public 
have necessitated performance improvement. Improved informa- 
tion about labor market obstacles and program effectiveness 
has contributed to a better understanding of how to serve current 
and future program participants. Yet at the same time, there 
has been concern over how to use performance standards to sup- 
port state and SDA programs and policies. 

The statistical adjustment models developed by the Department of 
Labor to set performance standards are but one part of the JTPA 
planning process. To the extent possible, the models adjust 
performance standards on the basis of variations among SDAs 
with respect to the characteristics of clients and the con- 
dition of the labor market. This model-derived performance 
level then becon>es the basis for establishing a standard, aft'sr 
considering the influence of other factors not accounted for 
by the model. States may further adjust the model-derived 
performance standard, or they may elect to use the standard 
derived through the adjustment model. 

The need to consider further adjustments to the statistical 
models can be seen from several different perspectives. From a 
technical perspective, some individual characteristics that 
influence employability , such as age, race, sex, and education, 
are included in the adjustment models and are factored into 
performance standards calculations. Others, however, such as 
basic skills and world-of-work knowledge are not now, but need 
to be considered in the standard-setting process. Moreover, 
unique service mixes and the structure of local labor markets 
are also important determinants of performance and require 
careful consideration in setting standards. 

To the extent that standards-setting is limited to the model- 
derived standards, program planners may limit their flexibility 
to respond to both participant and employer needs effectively. 
Performance standards-setting must be viewed as an interactive 
process that builds upon the statistical models by accounting 
for the expected influence of other important factors that bear 
upon an individual's status in the labor market and a program's 
ability to improve long-term earnings and potential. 
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From a state perspective, further adjustments to the statistical 
model can create a more flexible atmosphere for supporting state 
policxes and providing relief from what may be perceived as a 
constraining performance standards system. For example, a state 
may develop policies aimed at increasing services to the most 
difficult to serve, such as teenage mothers, in this case, 
adjustments to the model-adjusted performance standards may be 
warranted in those SDAs where such services were planned. 

In a similar vein, a state may develop coordination policies 
aimed at forging cooperative links between SDAs and state 
agencies responsible for serving particular target groups such 
as refugees. However, SDAs may be reluctant to serve such in- 
dividuals in large numbers without further adjustments to their 
performance standards. By making such adjustments, states may 
facilitate service provision to these groups and enhance coordina- 
tion. 

In the broadest sense, going beyond the model will provide states 
with substantially more opportunities to direct employment and 
training programs in a manner consistent with both their policies 
and priorities and the needs of prospective participants, it 
will allow them to support SDAs who wish to deviate from the 
existing standards and to take risks or new initiatives that 
otherwise may not be considered practical. 

It will provide SDAs a mechanism for more closely linking the 
employability and earnings of locally eligible participants with 
the types of programs that will ultimately be designed and 
delivered. 

This chapter provides background for a negotiated approach to 
setting JTPA performance standards by reviewing: (1) how the 
Department of Labor's adjustment model works and can be used to 
set performance standards; (2) the major limitations of the 
adjustment model; and (3) key issues involved in negotiating 
performance standards. 



THE DEPARTMENT OF LABOR'S ADJUSTMENT MODEL 



The Department of Labor has developed an optional statistical 
adjustment methodology to provide Governors with a systemmatic 
approach to conforming to the Secretary's parameters for setting 
SDA performance standards. 

The objective of the adjustment methodology is to statistically 
account for the influence of as many factors that affect 
performance as possible from the national level. 
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Early efforts to predict influences that affect performance 
revealed that even after controlling for client characteristics, 
program mix, and economic conditions, substantial variations in 
performance remained unexplained. While factors that could be 
readily quantified proved important, they were not all-mclusive. 
Some factors such as the quality of program management were 
excluded from the model. The basic goal is to hold local 
program operators harmless for factors over which they have 
little control such as client characteristics and labor market 
conditions. 

The current adjustment models do not account for all of the 
variation in local performance, leaving room for further 
adjustments. Across each of the existing adjustment models, 
between 12 and 40 percent of the variation of performance is 
explained in PY85 and up to 50 percent of the variation is 
explained in the PY86 models. Assuming that some portion of the 
variation could be attributed to the quality of management, 
the remainder may be due entirely to randomness or unmeasurable 
differences, or unmeasured factors that affect performance. 

Exhibit B lists the factors that have been included in the Program 
Year 1985 and 1986 adjustment models for the four adult measures 
and the three youth measures. As can be seen in the exhibit, 
depending upon the measure, the models include slightly different 
varieties of client characteristics and local labor market 
conditions. The objective of the adjustment models is to 
statistically estimate the effect of these factors on each of 
the performance measures and to use these estimates as weights 
for adjusting performance. 

Exhibit C presents an illustrative example of how these weights 
are used to set standards for PY86. As can be seen, the 
objective is to calculate the difference between an SDAs factor 
value and that for the nation. This difference is then mul- 
tiplied by the value of the individual factor weights derived 
from the statistical adjustment model. These weighted dif- 
ferences are then summed and added to or subtracted from what 
is referred to as the National Departure Point, which, in its 
most simple form, is the average national performance score 
in PY85, and the performance score adjusted to the 25th 
percentile in PY86. The end result is an SDAs model-adjusted 
performance level, which can be further adjusted to account for 
other important factors not accounted for by the model. 

Further adjustments to the model-derived performance standard 
are subject to the approval of the states. All states, however, 
must adjust program performance in accordance with parameters 
established by the Secretary of Labor (described in Chapter I) . 



26 



-16- 



EXHIBIT B 

Factors Included in DOL PY85 Adult and 
Youth Performance Standards Adjustment Models 



Local Factors Included in the PY85 Adult Worksheets 



Entered 
Employment Rate 



Cost per Entered 
Employment 



Average Wage 
at Placement 



TERMINEE CHARACTERISTICS: 



% 


Female 


% 


Female 


% 


Female 


% 


55 Years and 
Over 


% 


55 Years and 
Over 


% 


55 Years and 
Over 


% 


Black 


% 


Black 


% 


Black 


% 


Hispanic 


% 


Hispanic 


% 


Hispanic 


% 


Alaskan Native/ 
American Indian 


% 


Alaskan Native/ 
American Indian 


% 


Alaskan Native/ 
American Indian 


% 


Asian/Pacific 
Islander 


% 


Asian/Pacific 
Islander 


% 


Asian/Pacific 
Islander 


% 


Dropout 


% 


Dropout 


% 


Dropout 


% 


Limited English 
Speaking 


% 


Limited English 
Speaking 


% 


Limited English 
Speaking 


% 


Handicapped 


% 


Handicapped 


% 


Handicapped 


% 


Welfare Recipient 
at Entry 


% 


Welfare Recipient 
at Entry 


% 


Welfare Recipient 
at Entry 


% 


UI Claimant 


% 


UI Claimant 


% 


UI Claimant 



LOCAL ECONOMIC CONDITIONS: 



Average Wage for 
Area (000) 

Unemployment Rate 



Average Wage for 
Area (000) 

Unemployment Rate 

Average Weeks 
Participated 



Average Wage for 
Area (000) 

Unemployment Rate 
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EXHIBIT B 
(CONTINUED) 



Local Factors Included in the PY85 Youth Worksheets 



Entered 
Employment Rate 



Positive 
Termination Rate 



Cost Per 
Positive Termination 



TERMINEE CHARACTERISTICS! 
% Female 



% 55 Years 
and Over 

% Black 

% Hispanic 

% laaskan Native/ 
Americcux Indian 

% Asian/Pacific 
Islander 

% Dropout 

% High School Graduate 
and Above 

% Handicapped 

% Welfare Recipient 
at Entry 

% Single Head of 
Household 

LOCAL ECONOMIC CONDITIONS 

Average Wage for 
Area (000) 

Unemployment Rate 



% Female 

% 55 Years 
and Over 

% Black 

% Hispanic 

% Alaskan Native 
American Indian 

% Asian/ Pacific 
Islander 



% Handicapped 

% Welfare Recipient 
at Entry 

% Single Hp=td of 
Household 



Average Wage for 
Area (000) 

Unemployment Rate 



% Female 

% 55 Years 
and Over 

% Black 

% Hispanic 

% Alaskan Native 
American Indian 

% Asian/ Pacific 
Islander 



% Handicapped 

% Welfare Recipient 
at Entry 

% Single Head of 
Household 



Average Wage for 
Area (000) 

Unemployment Rate 

Average Weeks 
Participated 
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EXHIBIT B 
(CONTINUED) 



Local Factors Included in the PY86 Adult Worksheets 



Entered 
Employment Rate 



Cost per Entered 
E mployment 



TERMINEE CHARACTERISTICS J 



% Female 

% 55 Years and 
Over 

% Black 

% Hispanic 

% Alaskan Native/ 
American Indian 

% Asian/Pacific 
Islander 

I Dropout 

% Handicapped 

% UI Claimant 

& Welfare Recip- 
ient at Entry 



% Female 

% 55 Years and 
Over 

% Black 

% Hispanic 



% Asian/Pacific 
Islander 2/ 



% UI Claimant 

% Welare Recip- 
ient at Entry 



LOCAL ECONOMIC CONDITIONS: 



Average Wage for 
Area (OOO) 

Unemployment Rate 

% Families Below 
Poverty Level 2/ 

Population 
Density 

(lOOOs/sq.mi.) 2/ 



Average Wage for 
Area (000) 

Unemployment Rate 



Average Wage 
at Placement 



% Female 

% 55 Years and 
Over 

% Black 



Welfare Entered 
Employment Rate 1/ 



% Dropout 

% Handicapped 

% UI Claimant 

% Welfare Recip- 
ient at Entry 



Average Wage for 
Area (000) 



% Families Below 
Poverty Level 2/ 

Population 
Density 

(lOOOs/sq.mi.) 2/ 



% 55 Years and 
Over 

% Black 

% Hispanic 

% Alaskan Native/ 
American Indian 



% Dropout 

% Handicapped 



Average Weeks 
Participated 



Average Wage for 
Area (000) 

Unemployntent Rate 



Population 
Density 

(lOOOs/sq.mi.) 2/ 
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EXHIBIT B 
(CONTINUED) 

Local Factors Included in the PY86 Youth Worksheets 



Entered 
Employment Rate 

TERMINEE CHARACTERISTICS: 

% Female 

% 14-15 Years 

% 18-21 Years 2/ 

% Black 

% Hispanic 

% Alaskan Native/ 
Americcm Indian 

% Asian/Pacific 
Islander 

% Dropout 

% Student 2/ 



% Post High School 
Attendee 2^/ 

% Handicapped 

% Offender 2/ 

% Welfare Recipient 
at Entry 

LOCAL ECONOMIC CONDITIONS: 



Positive 
Termination Rate 



% Female 

% 14-15 Years 

% Black 

% Hispanic 

% Alaskan Native/ 
American Indian 



% Dropout 
% Student 2/ 



% Post High School 
Attendee 2J 

% Handicapped 

% Offender 2J 

% Welfare Recipient 
at Entry 



Cost per Positive 
Termination 



% Female 

% 14-15 Years 

% Black 
% Hispanic 



% Dropout 



% High School 
Graduate 2/ 

% Post High School 
Attendee IJ 

% Handicapped 

% Offender 2/ 

% Welfare Recipient 
at Entry 



Unemployment Rate 



Average Wage for 
Area (000) 

Unemployment Rate 



Average Wage for 
Area (000) 

Unemployment Rate 2/ 

Average Weeks 
Participated 



\J Adjustment models have been developed for this measure for the 
first time in PY 1986. 

2/ New factors in PY86 adult worksheets. 
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Illustrative Sample of Performance Standard Calculation 

(PY80) 



PY 86 JTPA performance Standards l*)rksheet 


A. Service Delivery Area's Nanve 


B* SDA Number 


C. perfonnance Period 


D. Type ot standard 


Date 


E. performance Measure 




PY 86 


[ J Plan 


Calculated 


E)itered Ehiployment Rate (/^ult) 




( ) Recalculated 









F. LOCAL FACTORS 


Gm SDA FACTOR 
VALUES 


H. NATIONAL 
AVERAGES 


1 

I* DlrFERENOE 
(G HlrUS n) 


J • WLilinio 
• 


K« LrrtjLi Ur UAJVL 
rLKruHnANLCi 
H TIMES J) 


1. % Finale 


59.0 


52.8 


6.2 


- .063 


Oft 


2. % 55 years old and over 


1.0 


2.9 


- 1.9 


- .079 


.15 


3. % Black 


45.5 


23.8 


21.7 


- .073 


- 1.58 


4. % Hispanic 


15.0 


7.9 


7.1 


- .022 


- .16 


5. % Alaskan Native/American Indian 


1.0 


1.3 


- .3 


- .010 


.003 


6. % AsiaVPacif ic islander 


5.1 


2.4 


2.7 


- .008 


- .02 


7. % Dropout 


30.2 


25.0 


5.2 


- .172 


- .89 


8. % Handica(^)ed 


11.4 


9.1 


2.3 


- .128 




9. % UC Claimant 


11.1 


10.9 


.2 


.014 


.003 


10. % Welfare Recipient 


42.0 


29.8 


12.2 


- .252 


- 3.07 


11. Average Wage for Area ($1000s) 


18.2 


16.9 


1.3 


- .653 


- .85 


12. Unenployment Rate 


9.0 


8.0 


1.0 


- .717 


- .72 


13. % Families Below Poverty Level 


6.2 


9.4 


- 3.2 


- .223 


.71 


14 • population Density (lOOOs/sq.ro.) 


7.0 


0.6 


6.4 


.827 


5.29 








L. Total 


- 1.8 


Source: U.S. Deoartment o£ Labor « Office of 


M. NATIONAL DEPARIURE POINT 


+ 62.4 


Strategic Planning and Policy Development; 
Guide for Settinq JTPA Title IIA Performance 
Standards for PY 86, Washington, D*C. 


L, 


N. Model-Adjusted Perfonnance 
Level (L + M) 


60.6 


NOTE: This worksheet is taken from the PY86 DO] 
dnustment model. Factors and weights differ be- 


0. Governor's Adjusbnent 


To be determined 



Lt performed in the same maimer. 


P. SDA Performance Standard 
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Although states are not required to use the statistical adjust- 
ment model for setting SDA performance ^ there are several 
compelling reasons for doing so. First/ the model provides the 
most objective means for accounting for the influence of readily 
quantifiable factors on local performance. Second/ the model 
introduces a high level of consistency and equity since all SDAs 
both within and across states can be evaluated using the same 
methodology and data. Third/ the model neutralizes the positive 
and negative effects of a variety of factors on performance and 
allows SDAs to target their resources to those who need it 
most. Fourth and last/ the use of the model focuses attention 
on who JTPA is able to serve well and who it is not. The model 
is based on actual prior performance and inaicates the effects 
of certain demographic groups or economic conditions on perfor- 
mance • 

For exc-^ole/ the negative influence of women/ blacks and 
handicapped/ on the entered employment rate suggests that added 
attention must be given to these groups to serve them suc- 
cessfully in programs. The same is the case for several other 
demographic groups such as high school dropouts. ThuS/ use of 
the model facilitates adjustments of performance standards 
based on target populations and program approaches. 



MAJOR LIMITATIONS OF THE ADJUSTMENT MODEL 



Although the adjustment methodology designed by the Department of 
Labor to establish performance standards has many advantages/ it 
also has some limitations. These limitations define the bases 
for negotiation and further adjustments to the model-derived 
performance levels. Each is discussed separately below. 



Selected Subgroups and/or Certain Terminee 
Characteristics are not Accounted for by the Model 



This model limitation is actually two separate issues — one 
dealing with groups of participants/ and the other with indi- 
vidual teinninee characteristics: 



Unmeasured Differences Among Program Terminees 
are not Accounted for by the Model 



Individually/ certain terminee characteristics which affect 
perfoinnance are not accounted for by the model. Some character- 
istics of individuals affect their potential to work and earn 
an income. On one hand/ demographic characteristics such as 
age, sex, and race capture the effects of employer preferences 
for different types of employees. On the other hand/ human 
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capital attributes such as education ^ experience, health, basic 
skills, and training measure the effects of the quality of labor 
on both the likelihood of employment and the type of employ- 
ment defined in terms of earnings and promotional opportunity. 
When attempting to explain differences in individual earnings 
and employment levels, it is important to account for as many 
of these characteristics as possible. 

Because statistical adjustment models are limited to data that 
can be collected for all SDAs and that which can be reasonably 
quantified, they primarily account for a standard set of demo- 
graphic characteristics (e.g., age, race, sex) and only a 
limited set of human capital attributes. Factors such as basic 
skill levels, health, and experience are not included because 
they are difficult to measure and data are not available on a 
national basis. This does not mean that they are unimportant. 
Rather, studies have shown that these factors contribute to the 
employment and earnings of participants. As a result, they 
should be accounted for in a performance standards framework 
provided that they are measurable and supported by relevant 
data. (See note 1) . 

For example, a young high school dropout with limited work 
experience and basic skills in reading and arithmetic will be 
difficult and relatively expensive to serve because he/she has 
limited functional literacy and little if any work experience. 
While the adjustment model will account for educational status 
and age, it will not fully capture the person's readiness to 
participate successfully in a program. It could be that to 
serve this youth, a program must make a substantial investment 
to help overcome the range of barriers faced in the labor market. 
Since the model only accounts for a portion of these barriers, 
it may predict too high an entered employment rate, and a cost 
level well below that necessary to make a pennanent improvement 
in the employability and earnings of this youth. 

Most JTPA administrators and planners can readily list impor- 
tant characteristics of their participants that affect labor 
market success but are not controlled for in the adjustment 
model. Many of these may be partially controlled for by other 
factors in the model. In many cases, however, data are not 
available to correct for the problem. First, many determinants 
of employability and earnings are not readily measurable, such 
as motivation. Second, data are not consistently collected on 
the state and local levels. Third, even if both of these issues 
could be resolved, it is difficult to determine the degree of 
independent influence of these additional factors on program 
performance. If states and SDAs are to overcome this short- 
coming of the adjustment models, substantial attention must be 
given to additional data collection and deliberation over the 
extent to which performance will have to be further adjusted. 



34 



-23- 



Selected Subgroups of the Terminee 
Population are Excluded from the Model 



Various groups have historically exhibited different prospects 
for future employment and earnings. Data show that men fare 
better than women; young adults fare better than older adults; 
and whites fare better than non-whites. The models used to 
adjust SDA performance factors account for many of these 
demographic characteristics. Other important terminee character- 
istics and local economic conditions that affect labor market 
success are not^ however^ included. (See note 2). 

Teenage mothers and displaced homemakers are examples of groups 
that face distinct barriers to successful labor force partici- 
pationr but are excluded from the adjustment models. Although 
JTPA is to be targeted upon these and other groups ^ SDAs and 
PICs determine most in need of services ^ the exclusion of these 
subgroups from the adjustment model may create disincentives to 
serve them. Their service needs are among the greatest ^ their 
likely service costs are among the highest ^ and their influence 
on program performance is likely to be negative. To serve 
them in large numbers may thus represent a risk that is too 
high for many administrators, given JTPA's emphasis on perfor- 
mance and the levels required to be eligible for receipt of 
six percent incentive funds. 

Addressing this limitation of the adjustment model is similar 
to the case of unmeasured differences among participants. 
While most JTPA practitioners can easily cite target groups 
who are excluded from the model, it is more difficult to specify 
the degree to which their exclusion actually affects perfor- 
mance. This issue will require added attention if further 
adjustments are to be made to the model -predicted performance 
scores . 



Youth Employment Competencies May Not 
Be Adequately Accounted for by the Model 

In developing adjustment models for the positive termination rate 
and the cost per positive termination, considerable concern 
surfaced regarding their validity. While the attainment of youth 
competencies has been included as a positive termination in 
PY84/85, they are not reported separately in the Job Training 
Partnership Act Annual Status Report (JASR) for the same 
period. (See note 3). For Py85, the attainment of youth 
employment competencies is not included in the youth models. 
As a result, the youth measures of the positive termination rate 
and its cost is driven primarily by the entered employment rate, 
and any adjustment models may misrepresent the true relationships 
between the factors included in the model and this performance 
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measure. The models for PY85 then do not adequately represent 
youth who may have other goals than entering employment. 

Youth competencies are, however, an important program outcome 
and these are reflected in the DOL PY86 adjustment models. 
Youth employment competencies focus upon basic and job-related 
skills, partly capture what was referred to earlier as 
"unmeasurable terminee characteristics," 

The PY86 adjustment models account for the attainment of youth 
employment competencies to some degree. The PY85 JASR did not 
have a separate item for youth employment competencies. 
However, the Job Training Longitudinal Survey (JTLS) — a survey 
of 192 SDAs — does collect this information. By using the JTLS 
data on youth employment competencies, the Department of Labor 
has derived the positive termination rate and cost measures for 
PY86. 

The information from the JTLS does not specify how lUany of the 
192 SDAs had reported youth employment competencies and whether 
those SDAs were representative. Therefore, although youth 
employment competencies are now accounted for in the PY86 
adjustment model, some adjustments (upward for instance on 
positive termination rate and costs) may still be necessary 
for those SDAs that have a fully developed competency system 
in place. For SDA's that are still developing a system, the 
PY86 models may suffice, or depending on program designs, 
positive termination rate may need to be adjusted downward. 



Differences in the Structure of Local 
Labor Markets are Not Accounted for in 
the Average Wage Adjustment Model 



Wages of an area are determined by several factors, including 
its geographic boundaries and mix of industries and occupations. 
For example, workers with similar skills may earn less if they 
work in a rural rather than an urban area. Similarly, wages for 
comparable skills vary across industries as well as occupations. 
To the extent that an SDA has a disproportionate share of certain 
types of industries or to the extent that only a segment of 
industries or occupations are within the practical reach of JTPA 
participants, its wage expectation may well deviate from the 
area average. 

The 1985 adjustment model used to calculate the wage standard 
for Title II adult programs included two labor market variables, 
the unemployment rate, and average area wage. Although both 
variables account for substantial variatio-i in the placement 
wages of SDAs, they may not always accurately reflect the 
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placement wage potential of an area. Two additional variables 
were introduced in PY86 — percent of families with income below 
the poverty level and population density. 

^urtheirmore^ the types of jobs available to JTPA participants 
may r^.present only a sub-set of all jobs in a local area. If 
these jobs pay less than the average wage^ then further adjust- 
ments may be required. 



Extreme Service Levels Produce Less 
Precise Model Performance Estimates 



The statistical adjustment models for PY85 and PY86 are based 
upon average SDA performance. In certain instances the models 
may predict performance expectations that are extreme in value 
when compared to those of other SDAs. These may be legitimate 
expectations given the conditions faced by the SDA. In other 
instances^ SDA factor values (e.g.^ percent female^ or the 
unemployment rate) may also be so large or small as to be deemed 
extreme in value. In these cases^ it may be necessary to make 
further adjustment to the model-derived performance levels 
particularly if more than one factor is "considered extreme." 

In its Program Year 1985 and 1986 Guide for Setting Title II-A 
Perfo3nnance Standards y the Department of Labor identified those 
instances under which either factor values or model-derived 
performance scores or both may be considered extreme. The 
extreme model-derived performance scores and factor values 
presented in Exhibit D are taken from these guides and may be 
used by states and SDAs to determine whether or not additional 
adjustments will be necessary. The model-derived performance 
scores indicate the range within which 98 percent of the SDAs 
expected performance scores would fall given the distribution 
of SDA factor values that existed in transition year 1984 
(October 1, 1983 through June 30, 1984) for Program Year 1985, 
and Program Year 1984 (July 1, 1984 through June 30, 1985) for 
Program Year 1986. The extreme factor values reflect the lowest 
one percent and the highest one percent values as reported by 
SDAs in the most recently available reporting period. 

Unexpected Events are Not Accounted 
For in the Model 



Unexpected events in local labor markets can either positively 
or negatively affect program performance. If, for example, a 
large employer closes during the program year, it may diminish 
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EXHIBIT D 

PY85 Extreme Model-Derived Performance Scores and Extreme 
Values of Local Factors for Adult and Youth Measures 



PY85 EXTREME MODEL-DERIVED PERFORMANCE SCORES 



EXTREME VALUES 



ADULT STANDARDS 
Entered Employment Rate 
Welfare Entered Employment Rate 
Cost per Entered Employment 
Wage at Placement 

YOUTH STANDARDS 
Entered Employment Rate 
Positive Termination Rate 
Cost Per Positive Temination 



EXTREMELY LOW 
Less Than 40% 
Less Than 21% 
Less Than $1,925 
Less Than $3,50 

Less Than 10% 
Less Than 60% 
Less Than $1,075 



EXTREMELY HIGH 
More Than 71% 
More Than 72% 
More Than $5,980 
More Than $5.65 

More Than 57% 
More Than 86% 
More Than $5,800 



PY85 EXTREME VALUES OF LOCAL FACTORS FOR ADULT MEASURES 



LOCAL F7\CT0R 














Percent Female 


Less 


Than 


26% 


More 


Than 


78% 


Percent 55 Years Old and Over 








More 


Than 


20% 


Percent Black 








More 


Than 


87% 


Percent Hispanic 








More 


Than 


84% 


Percent Alaskan Native/American 
Indian 








More 


Than 


15% 


Percent Asian/Pacific Islander 








More 


Than 


30% 


Percent Dropout 


Less 


Than 


6% 


More 


Than 


52% 


Percent Limited English Speaking 








More 


Than 


32% 


Percent Handicapped 








More 


Than 


28% 


Percent UI Claimant 








More 


Than 


33% 


Percent Welfare Recipient 


Less 


Than 


5% 


More 


Than 


67% 


Average Wage for Area (000) 


Less 


Than 


11.8 


More 


Than 


22.5 


Unemployment Rate 


Less 


Than 


4% 


More 


Than 


18% 


Average Weeks Participated 


Less 


Than 


4' 


More 


Than 


28 
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EXHIBIT D (CONTINUED) 
PY85 EXTREME VALUES OF LOCAL FACTORS FOR YOUTH MEASURES 

EXTREME VALUES 

LOCAL FACTOR EXTREMLLY LOW EXTREMELY HIGH 



Percent 


Female 


Less 


Than 


28% 


More 


Than 


70% 


Percent 


14-15 Years Old 








More 


Than 


•J ^ ^ 


Percent 


Black 








More 


Than 


94% 


Percent 


Hispanic 








More 


Than 


90% 


Percent 
Indian 


Alaskan Native /American 


— 






More 


Than 


14% 


Percent 


Asian/Pacific Islander 


— 






More 


Than 


32% 


Percent 


Handicapped 








More 


Than 


43% 


Percent 


Dropout 














Percent 


High School Graduate 














Percent 


Welfare Recipient 








More 


Than 


90% 


Percent 


Single Head of Household 


Less 


Than 


1% 


More 


Than 


32% 


Average 


Wage for Area (000) 


Less 


Than 


11.8 


More 


Than 


22.5 


Unemployment Rate 


Less 


Than 


4% 


More 


Than 


18% 


Average Weeks Participated 


Less 


Than 


4 


More 


Than 


27 
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EXHIBIT D (CONTINUED) 



PY86 EXTREME MODEL-PREDICTED PERFORMANCE SCORES 



ADULT STANDARDS 
Entered Employment Rate 
Welfare Entered Employment Rate 
Cost Per Entered Employment 
Wage at Placement 

YOUTH STANDARDS 
Entered Employment Rate 
Positive Termination Rate 
Cost Per Positive Termination 



EXTREME VALUES 
EXTREMELY LOW EXTREMELY HIGH 



Less Than 47% 
Less Than 43% 
Less Than $2,300 
Less Than $3.50 

Less Than 21% 
Less Than 65% 
Less Than $2,100 



More Than 77% 
More Than 60% 
More Than $6,400 
More Than $5.70 

More Than 62% 
More Than 86% 
More Than $5,300 



PY86 EXTREME VALUES OF LOCAL FACTORS FOR ADULT MEASURES 



LOCAL FACTOR 
Percent Female 

Percent 55 Years Old and Over 

Percent Black 

Percent Hispanic 

Percent Alaskan Native/American 
Indian 

Percent Asian/Pacific Islander 

Percent Dropout 

Percent Handicapped 

Percent UI Claimant 

Percent Welfare Recipient 

Average Wage for Area (000) 

Unemployment Rate 

Average Weeks Participated 

Population Density* 

Percent of Families Below 
Poverty Level 

PY86 EXTREME VALUES OF LOCAL 



Less Than 28% 



More Than 78% 
More Than 14% 
More Than 89% 
More Than 84% 



Less Than 7% 



Less Than 5% 
Less Than 12 
Less Than 3% 
Less Than 6 



More Than 
More Than 
More Than 
More Than 
More Than 
More Than 
More Than 
More Than 
More Than 
— (Max. = 



17% 
28% 
51% 
30% 
28% 
69% 
24 
17% 
33 
= 7) 



Less Than 3% More Ihan 25% 

FACTORS FOR YOUTH MEASURES 



LOCAL FACTOR 

Percent Female 

Percent 14-15 Years Old 



Less Than 31% 



More Than 71% 
More Than 34% 
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EXHIBIT D (CONTINUED) 

EXTREME VALUES 



LOCAL FACTOR 

Percent Black 

Percent Hispanic 

Percent Alaskan Native/ 
American Indian 

Percent Asian/Pacific Islander 

Percent Handicapped 

Percent Student 

Percent Dropout 

Percent High School Graduate 
or More 

Percent Welfare Recipient 

Percent Single Head of Household 

Average Wage for Area (000) 

Unemployment Rate 

Average Weeks Participated 



EXTREMELY LOW 



Less Than 4% 



Less Than 13% 



EXTREMELY HIGH 

More Than 95% 
More Than 93% 

More Than 16% 
More Than 25% 
More Than 50% 
More Than 76% 
More Than 59% 

More Than 79% 
More Than 60% 
More Than 31% 



Less Than 6 More Than 32 

PY86 EXTREME VALUES OF LOCAL FACTORS FOR WELFARE MEASURES 



LOCAL FACTOR 

Percent 55 Years Old and Over** 

Percent Black** 

Percent Hispanic** 

Percent Alaskan Native/American 
Indian 

Percent Dropout 

Percent Handicapped** 



Less Than 7% 



More Than 6% 
More Than 96% 
More Than 80% 

More Than 22% 
More Than 61% 
More Than 26% 



*Because population density is capped, there are no extreme 
values at the high end of the distribution. 

**Where local factors have extreme values at only the high end 
of the range, any value below this will be considered reasonabl . 
No range indicates that there are no values considered to be extreme. 
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anticipated employment opportunities for participants and 
adversely affect an SDA's ability to achieve its planned 
performance standards. In contrast, the completion of new 
development projects midway through the program year can have a 
positive influence on planned performance. Because unex- 
pected events can affect program performance, they are legitimate 
Items for consideration when conducting the annual review of 
performance. 

Unexpected events are difficult to isolate and to relate to 
program performance. States and SDAs should be careful in the 
use of this item, and should ensure that appropriate documenta- 
tion IS provided. 



KEY ISSU ES IN PREPARING TO NEGOTIATE PERFORMANCE STANDARDS 

To this point, the performance standards adjustment models and 
their chief limitations have been reviewed. The strengths of 
the models have been contrasted with their major limitations in 
order to identify important areas that states and SDAs may wish 
to consider to further adjust the model-derived performance 
scores. 

A critical point is that program performance can best be viewed 
withm the context of employ ability development, where the 
full range of factors that can affect participants' future 
employment and earnings is taken into account when setting 
standards. While the existing adjustment model accounts for 
some factors, there are several that it does not address. Basic 
skills and job-related skills are among the factors not included 
m the model but which remain important when planning a service 
strategy for the economically disadvantaged. 

Prior to engaging in a process to further adjust performance 
standards, it is important to focus upon the policies and 
procedures that will be used to support a negotiated approach 
to setting program performance. The first such issue is 
identifying the areas upon which negotiations will be based. 
While the previous section identified several reasons which may 
be relevant for further adjusting performance standards, they 
may or may not be relevant to each state and SDA. States and 
SDAs must therfore begin by considering each of the limitations 
of the adjustment models, their applicability to local programs 
and performance, and the types and magnitude of effects that may 
occur. This developmental work will identify basic areas that 
states and SDAs will use as the basis for negotiating performance 
standards . 
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Secondr states and SDAs must determine how negotiation will 
be dociiraented and used to further adjust the model-derived 
performance levels. For example^ it may be relatively easy to 
agree that unmeasured differences among participants are affecting 
performance. It is less easy^ however^ to quantify and correct 
for thisr and its effect on plannned performance. 

Thirds states and SDAs must establish planning guidelines that 
will determine additional data and reporting requirements needed 
to support negotiation, as well as the roles and responsibilities 
of the parties involved in the negotiation. 

Finally, the negotiation process itself should be open, non- 
political, and unbiased. In this way, states may ensure that 
SDAs have been handled consistently and fairly. 
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CHAPTER IV 
The Negotiation Process 



INTRODUCTION 



In the training and employment field, negotiation occurs all 
the time. State staff negotiate with each other and their State 
Job Training Coordinating Councils to develop policies and 
programs as well as with SDAs to determine coordination plans 
and best approaches for effectively serving eligible partici- 
pants* Similarly, SDAs negotiate with service providers over 
the terms and conditions of contracts and about policies and 
procedures for serving participants. 

In the case of performance standards, negotiation serves as a 
bridge between the goal of achieving high levels of performance 
and the realities of providing services to the economically 
di sadvantaged . 

This chapter provides a discussion of important issues that 
states and SDAs will face when negotiating performance stan- 
dards. In instances where states elect to incorporate nego- 
tiation into performance standards setting, the issues reflect 
a set of decision points that must be resolved. This negotia- 
tion process will define the requirements for SDAs to follow 
when documenting the need for negotiation as well as the con- 
ditions under which proposals for further adjustments will be 
accepted . 

In those cases where SDAs decide to initiate negotiation, the 
issues presented in this chapter represent the major planning 
tasks they will have to address to prepare a clearly documented 
proposal for further adjustments to the model-derived performance 
standards . For both parties, negotiating expected levels of 
performance imis^be based on a well-documented case that shows 
how targeting > programmatic problems, and remedies affect 
performance outcomes . 

The issues discussed in this chapter include: 

(1) documenting the basis for negotiation, 

(2) developing tools for quantifying adjustments to 
the model-predicted performance scores. 
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(3) negotiation and its relationship to six percent 
incentive funds, 

(4) establishing a framework to support a negotiation 
process, and 

(5) state and SDA roles and responsibilities in 
establishing a negotiation process. 



DOCUMENTING THE BASIS FOR NEGOTIATION 



The first step in developing a negotiation process is to 
determine the areas upon which further adjustments to the model- 
derived scores will be based. While the previous chapter 
Identified several potential areas for negotiation (unmeasured 
differences among program participants, exclusion of population 
subgroups from the model, etc.), states and SDAs must determine 
which, if any, are most applicable to their own programs. This 
will entail careful documentation of the problem by the SDA and 
its relationship to program design and outcomes, as well as its 
prevalence across and within SDAs in a state. 

Exhibit E presents the key elements of this process by identify- 
ing the tasks necessary to identify and ultimately document 
each of the potential areas of concern that are not adequately 
accounted for by the model. States and SDAs may choose a 
documentation process to address their own unique circumstances. 



Targeting Flexibility 



The fact that certain participants characteristics are unmeasured 
(and perhaps unmeasurable) and the exclusion of selected popula- 
tion subgroups from the adjustment models can potentially 
create disincentives to serve more disadvantaged participants. 
Because certain terminee characteristics that affect employ- 
ability and earnings are not included in the adjustment model, 
predicted performance levels may not accurately reflect the true 
level of difficulty of providing services. SDAiS and program 
operators may, therefore, perceive that the risks of providing 
certain services to various population groups are too great to 
take in light of the connection between performance standards 
and incentive/sanction policies. 

The first step in identifying the existence of this type of 

problem is for SDAs to determine those groups of individuals 

who are either not served well in existing programs or not served 

at all. It is important to specify the particular characteristics of 
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EXHIBIT E 



Matrix of Planning Tasks Necessary to Identify and Address Model*ReIated Issues 



Program and 
Targeting 
Issues 

Groups not served 
and/or partici* 
pant characteristics 
not accounted for in 
the model 



PROGRAM INTENSITY 
Planned durations 
and unit costs of 
existing program 
services 



Planned program 
mix 



Status of youth com^ 
petency systems and 
relative emphasis of 
employment*-oriented 
programs and develop* 
ment*oriented programs 



Extreme factor values 
and/or model^predicted 
performance scores 

Unexpected events 
and thexr effect on 
performance 



Model 
Limitations 



Perceived limited 
targeting flexi^ 
bility 



Tasks 



II 



PROGRAM INTENSITY 
Does not account 
for quality and 
type of services 
provided 



No program mix 
variable 



Youth employment 
cCTBpetencies may 
not be adequately 
accounted for 



Extreae service 
levels produce less 
precise model per-* 
fomance estimates 

Unexpected events 
in local area un*- 
accounted for 



Determine the bar* 
riers these groups 
face in the labor 
market 



Determine insuf* 
ficiency of program 
duration and cost 
for serving current 
participants and/or 
those not served 



Determine deficiency 
of program mix for 
current clients or 
those not able to 
serve 



Identify the program 
and performance im* 
plications of the 
omission of youth 
competency attainment 
from positive termi* 
nation rates 

Determine reasons 
for poor fit between 
the model and SDAs 
performance 

Determine links 
between such events 
and planned perfor- 
mance 



Determine likely pro*- 
gram success rates in 
serving excluded 
groups and partici<» 
pants with unmeasured 
characteristics that 
influence performance 



Determine more desir* 
able program duration 
and costs and estab*- 
lish their consistency 
with participant needs 



Determine more desir* 
able program mix and 
establish its consis* 
tency with the needs 
of prospective clients 



Determine revised pro** 
gram mix aimed at at* 
tainment of youth com- 
petencies 



Determine implications 
for performance stan- 
dards 

Determine implications 
for performance stan- 
dards 



III 

Determine the rea- 
sons for not ser- 
ving such groups 



Determine targeting 
implications and 
service levels of 
new participants in 
light of revised 
program mix 



Determine targeting 
implications of re- 
vised program mix 
and service plans 
for new participants 



One option is to 
develop youth com- 
petency systems 



IV 

Project service 
levels to these 
groups and de- 
termine program 
design^ cost and 
performance 
standards impli- 
cations 



Determine cost 
and performance 
implications of 
potential pro- 
gram and/or 
targeting re* 
visions 

Determine cost 
and per for* 
mance implica* 
tions of poten- 
tial program 
and/or targeting 
revisions 

Determine impli- 
cations for 
positive termi- 
nation rate 
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participants that are not adjusted for in the model such as 
earnings potential ^ motivation^ functional literacy ^ and 
basic skills. 

The initial identification of these characteristics and/or 
subgroups will be sufficient for establishing this as the basis 
for further adjustments. It will also be necessary to link 
the unique sf "vice needs of these groups/characteristics to 
expected perfo3nnance levels which would have to be higher or 
lower than that for the average program participant. Importantly ^ 
the extent of this difference will have to be large enough to 
influence the SDA-wide performance standards levels. If changes 
in SDA-wide performance levels cannot be demonstrated r adjust- 
ments will be difficult to propose and for states to approve 
(and unnecessary! ) ♦ 

Thusy after identifying who is not served well or not served at 
ally SDAs must determine and document the special labor market 
barriers and service needs of these groups and demonstrate 
that they are substantively different from those of the average 
participant ♦ This would then lay the foundation for negotiating 
performance levels beyond those provided by the adjustment model. 

Nexty SDAs must determine the expected performance levels 
that could be achieved for these groups andf on the basis of this^ 
why they cannot be served within the existing performance 
standards and program mix. Several sources of infoinnation exist 
for identifying expected success rates of these "hard to serve" 
groups. One source is the local or state management infoinnation 
system. It is likely that over time^ states and SDAs have served 
the types of participants they believe are most difficult to 
help. By reviewing the experience that previous programs have 
had in serving these population groups ^ it is possible to 
estimate likely performance outcomes for them. 

Another source of infoinnation on expected performance is the 
experience of other programs in the community. The evaluation 
and program analysis literature that may be available can provide 
evidence regarding the effectiveness of alternative program 
strategies for the economically disadvantaged. Local school 
systems — which have access to student characteristics # perfor- 
mance r and dropout data — are another source of info3nnation for 
determining what can be expected of programs that attempt to 
serve very disadvantaged youth. 

Based on this infoinnationr the final step is to summarize the 
program design and performance standards implications of in- 
creasing service levels to these diff icult-to-serve participants. 
If, based on analysis, it can be documented that the partici- 
pants face unique barriers in the labor market ^ have special 



48 



-36- 



service needs that cannot be adequately addressed by the current 
program mix and within the model -derived performance standards, 
then the basis for further adjusting performance standards will 
have been demonstrated. 



Program Intensity 



Under ideal circumstances, the mix, duration and/or costs of 
program services offered to current and prospective participants 
would reflect the type and level of effort needed to improve 
their future employ ability and earnings. However, because of 
limited resources, tradeoffs rsust be made between unit cost 
expenditures and the types of participants served. 

The current adjustment models attempt to inject objectivity 
into this tradeoff by compensating for the positive and 
negative effects associated with serving particular types of 
individuals. States and SDAs may become constrained in the 
types and intensity of services they provide if they believe 
that they can serve only those groups included in the model. 

Although the two cost models adjust for the duration of 
participation, this adjustment provides only a partial solution 
to serving particular groups. One reason for this is that tne 
factor weight for weeks of participation in the youth model is 
relatively small. (With a value of 53.3 in PY85, programs are 
compensated only $53 for each additional week or participation.) 
Since this is an SDA-wide program average, it accounts for tne 
addition of long-term program interventions to the total mix or 
services and provides adjustment when programs are substantially 
lengthened for a majority of participants. However, it does 
not permit reorienting targeting and mix of youth services to 
those who have a variety of short- and long-term intervention 
needs . 

To illustrate this point, consider an average duration of 9 
months or 39 weeks. According to the PY85 adjustment model, 
this would add $2,067 to the national departure point of 
$3,362, for an average cost per youth positive termination of 
$5,429. However, an SDA planning this type of program mix and 
performance level would be only one percent of all SDAs in the 
country. According to Department of Labor data for the Program 
Year 1985 model, 99 percent of all SDAs served youth for less 
than 27 weeks. If SDAs wish to plan and deliver this longer 
type of program, they should carefully document the results of 
these programs. States may want to encourage these more costly 
programs because of longer term benefits to youth. To do so, 
states should consider allowing much higher costs than predicted 
by the model. In view of this, negotiation may well be necessary 
to support such an approach. 
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A similar, although less extreme situation, exists for adults 

i?.^^?'''^'^ r^^^ °^ the duration of paJ^ 

ticipation IS 113.3 for PY85-thus offering more relief when 
increasing the length of stay in the program. 

thf pUls^^nd.^^^ ^1^.^^^ duration for all adult programs in 
the PY85 model was 12 weeks, and 99 percent of all SDAs served 
. adults m programs that lasted for less than 28 weeks. 

As a result of this problem, states and SDAs may well face 

?nr?i=4.S°''^^''^'-''J\V^^i? planning their program mix, duration, 
and costs. To establish this program intensity issie as a 
basis for negotiation, five basic steps can be relied upon: 

Step One . Identify the current program mix in terms 
Of program type, planned duration of participation 
for each program type, the unit cost of program 
services, and planned program outputs (such as the 
entered employment rate and positive termination 
rate) . 

Step Two. Examine the relationship between current 
programs and the unmet needs of current participants 
and/or of those not served because of perceived 
model under-adjustments. 

Step Three. On the basis of the previous step, docu- 
ment the insufficiencies that exist in current pro- 
gram intensity. As before, this process will require 
a careful examination and documentation of the em- 
ployment and earnings needs of prospective partici- 
pants. 

Step Four. Revise the service strategy to overcome 
those barriers that otherwise could not be addressed. 

Step Five. With this revised program mix, specify 
or modify targeting objectives and identify the 
implications for achieving model-derived performance 
levels. 

If, based on these tasks, it were demonstrated that program 
planning and targeting were constrained and that a revised service 
strategy was necessary to address the needs of existing and/or 
prospective participants, the basis for negotiation will have 
been documented. 
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Connection Between Performance Standards and 
Attaininent of Youth Competencie"s 

With the passage of JTPA, youth competencies were recognized 
as legitimate and important program outcomes. While the ultimate 
goal of serving youth is the attainment of productive employ- 
ment, awareness of the world of work, basic education skills, 
job-specific skills, and work maturity are important components 
of a service strategy aimed at alleviating the problems 
disadvantaged youth face in th^ labor market. 

Since JTPA was implemented, youth competency performance data 
have not been collected as a separate line item on the JTPA 
Annual Status Report (JASR) . As a result, these data were not 
included in the positive termination rate model for PY85. 

Since the PY95 model does net include attainment of youth 
employment competencies, the total positive outcome attained by 
SDAs with a PIC-certif ied youth employment competencies program 
may actually increase the positive outcomes achieved depending 
upon the types of youth served. On the other hand, total cost 
may rise as well. An SDA will need to know not only how many 
youth terminated by attaining youth employment competencies, but 
how this affected the length of the program in comparison to the 
length of program for all youth served. 

The Department of Labor has encouraged states and SDAs to 
design and implement PIC-certif ied competency-based systems and 
to measure performance on the basis of these systems. This 
system will yield the benefits of increased targeting flexibility, 
increased program design latitude, and the ability to chart a 
carefully developed and comprehensive service strategy for youth. 
The PY85 positive termination rate and cost model — which does 
not include attainment of youth competencies—would have to be 
further adjusted by states to reflect the program design and 
targeting implications of a new or increased emphasis on youth 
employment competencies and on employability development-oriented 
programs. (Two approaches have been developed and are described 
in the Department of Labor's Technical Assistance Guide for 
Setting JTPA Performance Standards for PY85.) 

Once a competency-based system is in place, it will be necessary 
for SDAs to determine their programming and targeting implica- 
tions and to compare them with those of programs aimed pri- 
marily at job placement. The objective is to draw out the 
important differences between both and to use them as the basis 
for making necessary revisions to the job training plan. This 
then, will begin to identify how the positive termination rate 
and its cost should be revised. Costs may go up, but the 
positive termination rate may increase as well. If it can be 
shown that the presence of youth competencies affects participant 
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targeting, program design, or expected success on positive 
terminations, then it can be effectively argued that negotiation 
is necessary. ^ 

The DOL PY86 youth positive termination rate model accounts for 
the attainment of youth competencies. These data have been 
collected for PY86 in the Job Training Longitudinal Survey 
(JTLS) . It may be useful to use the PY86 model as part of the 
negotiation process for PY85. When recalculating its actual 
PY85 performance, an SDA could compare the 1985 and 1986 models 
for both costs and positive termination rate. The rate provided 
by the 1986 models can provide a numerical base line from which 
to begin negotiation. 

However, the 1986 model recalculated standards should not be 
viewed as a panacea. As discussed earlier in this section, how 
youth employment competencies affect program targeting and 
design is central to the negotiation. The 1986 model accounts 
for some of these decisions, but certainly not all. 

Another adjustment consideration needs to be addressed when 
discussing the 1986 model for positive termination rate. Many 
PICs have chosen not to have piC recognized competencies or 
have chosen not to use their systems for positive termination 
purposes, but rather for program design purposes, if the PiC 
does not have a system of youth competencies in place, the 1985 
models probably are sufficient for the purpose of determining 
standards for PY86. 

Whereas the concern in 1985 was to adjust for those SDAs that 
had youth employment competencies where the PY85 model did 
not account for this variable, the issue when using the 1986 
model will be how to account for not having a system of youth 
employment competencies. For SDAs without youth employment 
competencies, the 1986 model may predict a higher positive 
termination rate and higher cost. 



Adjustment Model for Average Wage Standard 
Does Not Account for Differences in the 
Structure of Local Labor Markets 



Since JTPA is intended to improve the employment and earnings 
of participants, programs are encouraged to achieve~maximum 
possible wages at the time of job placement. The average wage 
at the time of the job placement is intended to reflect the 
quality of jobs that states and SDAs are able to secure for 
their participants. 
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Placement wages are, however, influenced by several factors, 
including the characteristics of rlients served and the condition 
and structure of local labor markets* To the extent that dif- 
ferences in these factors may vary within and across SDAs, the 
modeling ability to produce a given wage at the time of place- 
ment will also vary. Consider, for example, two SDAs, one 
serving very large shares of women and the other serving very 
large shares of men. Since men typically earn more than women, 
we would expect average wages of program terminees to reflect 
this difference. Unless these and other differences between 
SDAs are controlled for, it will not be possible to compare 
their wage performance in a consistent and equitable manner. 

The statistical adjustment model used to set placement wage 
standards attempts to control for these differences in two 
ways. First, the model includes a series of variables designed 
to capture differences in the characteristics of clients served. 
These variables focus on age, race, sex, education, welfare 
status, and other related variables. The model also attempts to 
control for differences in the condition and structure of local 
SDA labor markets . 

The structure of the labor market is more difficult to measure 
because of differences in the nature and type of available jobs, 
wage paid for various jobs, etc. For example, since different 
industries and occupations pay different wages for the same 
type of worker, such differences across SDAs could influence 
the wage rate at which they place participants. Similarly, 
wages tend to vary between urban and rural areas for the same 
type of workers. Unless these and other differences in the 
structure of local labor markets are taken into account, 
estimating SDA perfoirmance measures may not be possible. 

The PY86 statistical adjustment model includes four variables 
designed to measure the condition and structure of the labor 
market. The first two variables are the unemployment rate and 
the percent of families with income below the poverty level. 
These variables represent prevailing local economic conditions. 
The latter two variables — the average area wage and population 
density— are designed to capture structural differences in SDA 
labor markets. These variables capture some, but not all, 
factors that affect the labor supply and therefore only partially 
represent the structure of local labor markets. 

The model has been criticized because it uses Bureau of Labor 
Statistics data which include all types of occupations including 
both high paying jobs such as for CEOs and low paying entry- 
level jobs. Some argue that the critical issue is how to adjust 
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the average wage data to now realistically represent the jobs JTPA 
trainees may be qualified to hold. Rural and urban wage dif- 
ferences, as well as variations in the types of occupations 
available to JTfA participants in various SDAs may require that 
additional adjustments be made to the model-derived wage standard 
for PY85. 

For PY86, the rural-urban issue has been addressed to some 
extent by the addition of the factor "population density." 
The nature of the adjustment that may be needed for FY 86 is 
related to occupations available to JTPA clients in either a rural 
or urban area. If the jobs available to JTPA participants are 
predominantly entry level, then an adjustment to the model- 
derived wage standard may be appropriate. (The real question 
is how the local labor market is so different from the national 
structure that it warrants an adjustment.) 



Extreme Service Levels Produce Less 
Precise Model Performance E s t ima t e s 



The statistical adjustment models are based upon average SDA 
performance. In certain instances the model may predict perfor- 
mance expectations that are extreme in value when compared to 
those of other SDAs. These may bo legitimate expectations 
given the conditions found by the SDA. In other instances, SDA 
factor values (e.g., percent female, or the unemployment rate) 
may be so large or small as to also be deemed extreme in value. 
In these cases, it may be necessary to make further adjustment 
to the model-predicted performance levels particularly if more 
than one factor is "considered extreme." 



Adjustments for Unexpected Events 



In training and employment programs, changes in local labor 
markets such as plant closings, the inability to recruit planned- 
for target groups, and the opening of new development projects 
are among the types of unexpected events that can potentially 
affect achievement of planned performance levels. Since these 
factors cannot be anticipated at the outset of the program year, 
further adjustments to performance levels must be done at the 
close of the program cycle. 

It should be stressed that although unanticipated events may 
be difficult to use as the basis fcr negotiation (especially 
when they occur within the realm of environmental factors) SDAs 
should carefully monitor the behavior of all unexpected events 
as they occur and document how, if at all, they are affecting 
planned performance. 
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For example, if sudden plant closings occur in a local area, 
to be considered as the basis for negotiation they will have to 
be clearly tied to the planned outputs and especially place- 
ments of the program and used to demonstrate why the established 
performance levels could not be achieved. 

An additional example would be the inability to recruit a 
planned for target group. This may adversely affect cost (if 
recruitment efforts were extensive) but other standards may 
be adjusted by the model itself if the groups actually served 
are not unusual. 



Summary 



Effective negotiation is based upon a carefully developed and 
informed process of documenting the employability needs of the 
eligible population and external factors that affect it. This 
process serves as the basis for negotiating performance 
beyond that predicted by the statistical adjustment model and 
can help to improve the effectiveness of programs for the 
economically disadvantaged. 

States wishing to take a negotiated approach to setting per- 
formance standards will have to design a set of policies aimed 
at documenting the need for further adjustments to the model- 
predicted performance scores. These policies will provide 
guidance to SDAs by identifying the requirements necessary to 
substantiate the need for negotiation. They will not specify 
how adjustitients are to be quantified, but will lay out the 
planning and development tasks that will be necessary before 
any further adjustments are to take place. 

SDAs may wish to take the initiative and propose adjustments to 
their model-predicted performance standards. In these cases, 
the first task will also be to document the need for such 
adjustments. Such documentation must clearly link the model- 
derived performance standards to the SDA's planned program 
and/or targeting goals and determine whether or not adjust- 
ments to the model-predicted performance scores are warranted. 



DEVELOPING THE TOOLS FOR NEGOTIATION 



Once the basis for negotiation has been established, states and 
SDAs can engage in the actual process itself. This will require 
that basic negotiating tools be developed and used to further 
adjust the model-predicted performance scores. These tools will 
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haye to be selected by the states as part of their negotiation 
policy or by SDAs who wish to initiate negotiation themselves. 

There are two basic types of tools that can be used to further 
adjust performance standards. 

The first type includes a number of methods for quantitatively 
adjusting performance standards once they are es timated by the 
statistical model. These methods are most desirable since they 
can be objectively and consistently applied to all SDAs and to 
all program operators. 

The second type of tool is qualitative in nature and adjusts 
performance on the basis of one's best understanding of success 
in the JTPA program, it is not a substitute for quantitative 
adjustments, but may have to be used if it is not possible to 
estimate the effect of a particular issue on performance. 

Exhibit F presents the available quantitative and qualitative 
adjustment tools that states and SDAs can elect to use and the 
areas of negotiation to which they can best be applied. The 
quantitative tools include: 

1. The use of additional performance measures 

2. Weighted averages of performance 

3. Weighted indices (a legitimate but risky tool) 

4. Adjustments to specific standards 

There is one qualitative tool, increasing the tolerance inter- 
vals, that would be used only if further adjustments could not 
be quantified. 

Following is a detailed discussion of each of the different 
adjustment alternatives and the issues to which they can best 
be applied. 



Use of Additional Performance Measures 



The use of additional performance measures is one quantitative 
tool for providing relief from what may be perceived as perfor- 
mance standards that constrain both targeting and program 
design decisions. If a state or SDA has determined that 
limitations to the adjustment model constrain its ability to 
serve certain types of participants or offer certain types 
of programs, it may consider the use of additional measures. 

Several states and SDAs have taken this course and have con- 
centrated on the use of additional input measures of succe ss. 
These measures have typically been relied upon to facilitate 



EXHIBIT F 

Established Areas for Negotiation and Potential Negotiation Tools 



Negotiation Tools 
Quantitative Qualitative 



Established Areas for 
Neqotiation 


Additional 

Performance 

Measures 


Weighted 
Averages 
of Perfor- 
mance 


Weighted 
Indices 


Adjustments 
to Specific 
Standards 


Increase 

Tolerance 

Levels 


1) Unmeasured character- 
population subgroups 




a 

w 


• 


9 


i 


2) Program intensity 
(duration* costs* mix) 




t 


• 


• 


• 


3) Positive termina- 
tions and youth em- 
ployment competencies 


• 


• 


• 


• 


f 


4) Average wage at place- 
ment vis a vis labor 
market differences 








• 




5) Extreme service levels 




• 






• 


6) Unexpected events 








• 


• 
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services to groups which would not otherwise be served or to 
increase the proportion of a given target group served. In 
this first instance, some states have already established so 
called "equity standards" to promote service to groups who may 
or may not be in the model in relation to their proportion in 
the eligible population. In the latter case, they have and 
may include measures that document the extent of services to 
particular population subgroups such as teenage mothers r dis- 
placed homemakers, and participants with limited functional 
literacy. 

Such measures would be developed by identifying the subgroups 
under question and establishing a service share measure* 

There is no reason that additional measures need to be con- 
fined to input measures. Output measures of success not 
already used are also viable options. Additional output measures 
are useful because they can potentially broaden the perspective 
on perfoinnance and hence increase both targeting and program 
design options. For example, it has been argued that the sole 
emphasis on termination-based measures limits the extent to 
which programs aim to produce lasting effects on the employ- 
ability and earnings of participants. If post-program measures 
were also used (prior to their planned use by DOL in 1988) 9 this 
would shift some of the planning emphasis into the labor market 
that participants enter once they leave a program, (See note 
4,) As a result y programs may become increasingly conscious 
of the type of designs that are necessary to help participants 
both obtain and retain employment:. 

Another alternative is to develop certain output measures for 
selected subgroups of the population. In this case^ rather 
than relying on input measures or program-wide output measures , 
additional performance measures can be developed to assess the 
effectiveness of services to groups that are not usually served. 
For example, placement rates or post-program retention rates 
can be used as measures of success for dir^laced homemakers # 
or those with limited functional literacy. 

In all cases when output measures are used^ states and SDAs 
will have to determine an acceptable level of performance. 
One option is to approach the problem the same way that the 
entered employment rate is calculated for adult welfare 
recipients; that is, using the ratio of the standard for the 
subgroup (assuming some past experience serving that group) 
to all groups considered. Another option is to review the 
level of success of the SDA or other programs in serving the 
particular subgroup and to use it as the level of expectation. 
In both cases, it is important to point out that they require 
estimates of the performance standard for the population sub- 
group under consideration. Other program experiences or a 
state's or SDA's management information system are possible 
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sources for these data. 

Once the additional measures are decided upon, it is necessary 
to determine their importance relative to the existing national 
measures as well as the relative importance of each of the 
national measures. Since new measures are being added to 
offset the documented effects of the existing national measures, 
it follows that they should receive relatively more impor- 
tance than at least one of the national measures. The most 
straightforward approach is to allow SDAs to meet some combina- 
tion of the total number of performance measures. 

The use of additional measures as a negotiation tool has certain 
advantages. First, it can broaden a program's targeting and 
design perspective by offsetting the documented effects of the 
model-derived performance standards. Second, additional measures 
can allow SDAs to take a more flexible approach to program 
planning. Third, if the use of additional measures is planned 
along with incentive funds distribution, it can serve to 
create consistency in the "messages" that states send to SDAs 
regarding priorities for the training and employment system. 

On the other hand, the use of additional measures does have 
certain disadvantages. For example, too many performance 
measures may diffuse and possibly confuse the purpose of 
providing JTPA services. Further, it is not prudent to 
establish additional measures for some SDAs within a state but 
not for others. Therefore, this approach is recommended only 
for statewide application. 



Use of Weighted Averages of Performance 



In the case of weighted averages, each performance measure 
becomes the potential object of negotiation and is adjusted 
to account for particular limitations of the model. The 
adjustments are made by weighting the model-predicted score 
by a value for the same measure determined for the particular 
population group the SDA does not believe is accounted for. 

Thus, an entered employment rate can be adjusted downward 
and its cost upward to account for services to participants 
with limited functional literacy. Weighted averages may be 
difficult to implement since the entire focus of the nego- 
tiation is upon one or a few standards and adjustment to them. 
Weighted averages are, however, a viable approach. 

As with each of the negotiation tools, the use of weighted 
averages would begin with a careful documentation of the 
principal limitation (s) of the model that is (are) to be 
addressed and the disadvantages it is causing. In this case. 
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the limitations would focus primarily upon unmeasured character- 
xstxcs and excluded population subgroups. 

Once the problem and its cause (s) are identified, it is 
necessary to determine which performance standard (s) are to 
be ad3usted and the expected value for the group (s) not 
accounted for in the statistical model. With the expected 
performance value of the particular group (s) calculated, it 
IS then possible to weight the overall performance measure with 
the following formula: 

(TT-NTGT) (PAT) + (NTGT) (PTGT) 

= WP 

TT 

Where: 



TT 
NTGT 



Total expected number of terminees 

Expected number of terminees in special group 

PAT = Performance standard for all terminees other than 
members of special group (e.g., EER from work- 
sheet using planned characteristics of regular 
terminees) 

PTGT = Expected performance for terminees in special 
group 

WP = Weighted performance standard for all terminees 

As an example, consider an SDA which believes that it cannot 
effectively serve disadvantaged young adults with limited 
education unless the model-predicted entered employment rate 
IS lowered. For the next program year, it plans to serve 100 
of these participants and place them in unsubsidized r^mploy- 
ment at a rate of 40 percent, it also plans to serve a total 
of 500 participants and must, according to the model-predicted 
score, place them at a rate of 70 percent, m this case, the 
weighted average calculation would work as follows: 



(500-100) (70.0) + (100) (40.0) 



= 64.0% 



500 



As can be seen, using a weighted average lowers the model- 
derived placement rate from 70 percent to 64 percent. This 
basic approach can be used to adjust any of the seven national 
performance standards. 

Weighted axerages can also be used to adjust the positive 
termination rate and its cost, particularly for PY85 when youth 
employment competencies are not in the model, in this case, the 
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objective may be to reduce the model-predicted positive 
termination rate since it is so largely determined by the 
entered employment rate. This would be accomplished by weight- 
ing the model-derived performance score by the share of 
positive terminations that are planned to attain youth com- 
petencies. The benefit of this would be to divert emphasis 
from placement-oriented programs to developmental programs 
that emphasized the achievement of specific competencies 
that youth need to succeed in the labor market. The following 
formula would be used: 

(YEER) (SEP) + (AYC) (SEEP) = WPTR 

Where : 

YEER = Youth model-predicted entered employment rate 

SEP = Share of youth to be served in employment 
oriented programs 

AYC = Attained youth competency rate 

SEEP = Share of youth to be served in employ ability 
enhancement programs 

WPTR = Weighted Positive Termination Rate 

The most critical issue when using weighted performance stan- 
dards is the ability to determine the expected performance 
level for the particular group in question. For states and 
SDAs who have had experience serving such groups, the task 
may not be so difficult since they can retrieve performance 
data from their management information systems. In other 
cases, however, states and SDAs will want to consult other 
sources such as other local programs, or comparable SDAs for 
the information, in order to ensure that low performance for 
special groups is not inadvertently promoted. 

This problem, however, will be most evident during the first 
year of implementation, since at the end of the program year 
and in future years, data will have been accumulated to 
measure performance for the groups that consistently receive 
most attention. Thus, the risk of making a mistake can 
largely be confined to the initial year of implementation as 
long as necessary data collection takes place. 

Moreover, as long as the adjustment formula is in place, 
performance standards are re-calculated at the end of the 
program year when actual data are available. And, assuming 
that SDAs do not lose sight of achieving their standards for 
all terminees, they run only a small risk of under performance 
on account of serving a relatively small number of participants 
with unique barriers in the labor market. 
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The SDA could negotiate a planning standard using this method 
even when no data are available. At the end of the year, if 
the SDA has served the planned target group as negotiated, the 
negotiated adjustment to the standard would apply, if the 
SDA failed to serve the number of individuals in the target 
group or failed to offer the type of service, a new standard 
would have to be negotiated. 



Use of Weighted Indices of Program Performance 



Weighted indices of performance represent one tool for allowing 
states and SDAs to vary the emphasis on different measures in 
order to best reflect their particular priorities. These 
indices can be constructed with the existing performance 
measures or with additional measures that a state and SDA may 
wish to add. Weighted indices are similar to the approach 
many states currently use which weights individual standards 
to emphasize different policy objectives. The advantage of 
using an index of performance is that all performance measures 
can be treated as part of an overall approach to performance 
rather than treating each of the Secretary's seven measures 
(and any others the state adds) as separate measures. The 
weighted index takes all measures and develops an index by 
adding all the observations of the outcome measures (for 
example, entered employment, average wage at placement, wel- 
fare entered employment for adults) and dividing that total by 
the applicable cost measure, since measures for adults and 
youth are somewhat different, and since youth and adult 
programs cost may vary significantly, it may make sense to use 
separate adult and youth performance indices. 

The use of weighted indices of program performance could be 
tied directly to the state s incentive award policy, or use 
of such indices could exist independently of that policy to 
determine if performance standards have been exceeded. 
Weighted indices are simply a composite approach to judging 
overall performance. The tie to incentive awards could be 
made easily. Using an index assumes that the greater the 
ratio attained over "1" the larger the share of the incentive 
funds an SDA might be eligible to receive. 

The weighted indices approach suggested could be used in two 
main ways: 

(1) The first way would be to use a uniform statewide 
approach so that the weight for each measure with- 
in the index would be the same for all SDAs. This 
is similar to the approach many states now use 
where each measure may have different weights. 
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but the same weights are uniformly applied to 
all SDAs. For example ^ a state might choose to 
give the adult measure and youth measures 
equal weight so that incentive funds are initially 
divided into two equal pots of money. For the 
adult portion r the state might choose to empha- 
size the attainment of the entered employment 
rate standard and weight it 50 percent. To em- 
phasize increasing earnings r the state might 
choose to weight the average wage at placement 
30 percent r and finally ^ the welfare entered 
employment rate might have a weight of 20 per- 
cent. (See Exhibit G for an example of how this 
would work in an index.) The SDA whose ratio on 
the index is greater than "1" would be con- 
sidered a superior performer. This has the 
effect of giving strong emphasis to state-directed 
policies. 

(2) The second method would be to develop individual 
weights for each measure for each SDA. This 
would entail negotiating individual weights for 
each measure with each SDA depending upon their 
individual circumstances. For example ^ an SDA 
might want to trade-off one measure against 
another so that one measure had more weight than 
another. This might be the case particularly 
if they believed that the model was not 
adequately representing their situation. This 
could apply whether the state used only the 
Secretary's seven measures or whether additional 
measures were used. This individualized approach 
also gives states the opportunity to negotiate 
different eraphases among standards for each SDA . 
If the state believed that one SDA needed to 
improve service to welfare recipients r they 
could negotiate a greater weight for this 
standard* 

It is important to note that while varying the weights of 
existing or additional measures provides maximiim flexibility ^ 
it can become quite difficult for a state to negotiate dif- 
ferent weights separately with each SDA in a uniform^ equitable y 
and consistent manner. The possibility exists that states 
would be vulnerable to complaints and/ or legal action from 
SDAs who believe they were unfairly treated. It should be 
further noted that the Department of Labor's Employment and 
Training Administration does not endorse the individualized 
SDA approach to weighted indices y as it removes the protections 
afforded by the model. As a result ^ states should carefully 
consider the innovative benefits and possible drawbacks of 
weighting existing or additional measures using indices * 
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EXHIBIT G 



Illustrative Example of an Additive, Weighted Index of Title IIA 
Adult Program Performance for Two SDAs 



(A) 

Performance 
measure 



(B) 
Assigned 
weight of 
the mea- 
sure 



SDA I 



SDA II 



(C) 
Achieve- 
ment 
rate 



(D) 
Achieve- 
ment rate 
X weight 
of measure 



(C) 
Achieve- 
ment 
rate 



(D) 
Achieve- 
ment rate 
X weight 
of measure 



Entered un- 
subsidized 

employment .50 

Average wage 
at placement 
standard .30 

Welfare 
entered 
employment 

rate ,20 



90 



60 



120 



45.0 



18.0 



80 



120 



40.0 



36.0 



24.0 



120 



24.0 



Total value 
of index of 
performance 

Value of in- 
dex of pro- 
gram costs 

Value of Program Index 
Value of Index of Costs 



87.0 
83.3 



87.0 



83.3 



1.04 



100.0 
125.0 



100.0 

125.0 
80.0 



Definitions: 

(1) Entered Unsubsidized Employment: The ratio of the number of par- 

ticipants placed in unsubsidized jobs upon termination to total 
number of terminations from the program. 

(2) Average Wage at Placement Standard: *Mean beginning hourly wage of 

unsubsidized job placements divided by SDA wage standard times 100* 

(3) Value of Index of Program Costs: Cost of program per unsubsidized 

entered employment divided by average cost per unsubsidized place- 
ment established as a standard times 100. 
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However/ the organizations which reviewed and contributed to 
this guide believe that the use of weighted indices is a 
viable approach which may have the benefit that it can sim- 
plify how success against standards is viewed and potentially 
simplify the incentive award system. 

For those states that wish to consider the use of weighted 
indices, examples and the steps necessary to construct a 
weighted index of performance are proviced. 

In order to construe*" a weighted index of performance the 
following basic steps would be necessary: 

1. Identify all measures to be used to assess per- 
formance (including additional ones identified 
as important but not included among the seven 
national treasures) ; 

2. Deteirmine the model-derived or actual perfor- 
mance standard for each of the measures (depen-^^ 
ding upon whether the index is being used for 
planning purposes or to recalculate standards 
at the end of the year) ; 

3. Develop relative weights for each of the measures 
that total to 100 percent; or vary the relative 
measures for each SDA. (See Column B in Exhibit 
G); 

4) Convert the model-derived standard to an "achieve- 
ment rate." This is a percentage of the model- 
derived standard which is either planned to be 
achieved (if planning figures are used) or is 
actually achieved (if end-of-year actual, recal- 
culated standards are used) . (See Column C in 
Exhibit G) ; 

5) Multiply each of the performance standards by its 
weight to determine a factor value. (See Column 
D in Exhibit G) ; 

6) Sum the factor values to estimate the percentage 
value of the performance index for the rate 
measures and then divide by the percentage of 
cost standard attained to determine the com- 
posite planned performance level; 

7) At the end of the year, recalculate the composite 
index r using actual data to determine what percen- 
tage of the model-derived standard was achieved. 

Exhibit G presents an illustration of how a weighted index of 
performance would work for Title II-A adult programs. In 
this example, the state has decided to rely only upon the 
existing national standards. 
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As can be seen in Exhibit G, the entered employment rate has 
been given more weight than the other two adult measures. 
Since all measures in a weighted index approach need to be 
expressed in terms of percentage, the wage standard is con- 
verted to an achievement rate by planned level of performance 
by the model-derived score (at the beginning of the year) and 
the actual wage by the re-calculated model-derived score (at 
the end of the program year) . This same type of procedure 
would be necessary for any other performance measures that 
are not expressed in percentage terms. 

At the outset of the planning process, the value of these 
"achievement rates" may be equal to, less than, or greater 
than the- agreed upon performance standard level. If this 
level is set on the basis of the model-derived performance 
score, then it will equal 100.0. If, however, setting of 
the standards can occur within the tolerance interval established 
by the model, then the value can exceed or be less than 100.0. 

Once each of the multiplications are complete, the products 
would be summed to calculate the total value of the perfor- 
mance index. This index is then divided by a cost index 
which is simply the ratio of the model-derived cost per 
entered employment to the final cost standard. The result is 
a total perfoirmance score that incorporates all performance 
standards into one measure that best reflects the priorities of 
the state and its SDAs. This comprehensive score then be- 
comes the performance standard that is calculated at the begin- 
ning of the year on the basis of planned data and at the end of 
the year on the basis of actual data. 

In the present example, SDA II outperforms SDA I in terms of 
its performance index, owing largely to high achievement on the 
average wage at placement standard and the welfare entered 
employment standard. However, once the index of program 
costs is accounted for, SDA I outperforms SDA II because of 
its higher" levels of efficiency. As should be obvious, the 
assignment of different weights to each performance standard 
can hav? a marked effect on an SDAs performance. 

On the basis of this example, it should be clear that the 
assignment of weights to each performance measure can provide 
a useful and important tool in negotiating performance and 
influencing program targeting and design. In the above case, 
SDA II would be forced to reduce its costs. With the oppor- 
tunity to negotiate performance measure weights, each SDA can 
compensate for the limitations of model-derived performance 
scores by the relative importance that will be given to dif- 
ferent measures. States can use the index to address perfor- 
mance problems. 
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If a state decides that a weighted index of performance will 
be used as its tool for negotiation ^ it will also have to 
determine in what form to implement the weighting scheme. 
Five alternative options are discussed below: 

1. The first option is to limit the index to the 
existing Department of Labor measures and assign each 
one a uniform statewide weight as shown in Exhibit G. 
These weights would be pre-determined by the state 

or developed in consultation with SDAs. The 
advantage of this approach is its simplicity. New 
measures and standards would not have to be de- 
veloped and individual negotiation with SDAs over 
relative weights would not be necessary. More- 
over r the approach would provide some relief from 
the model-derived score since some standards could 
be weighted more heavily than others. 

The drawback to this approach is that it may not 
fully account for the unique circumstances faced 
by any particular SDA. Since both the performance 
measures and their weights are fixed r the type of 
"further adjustment" needed is assumed to be the 
same across each SDA. 

2. The second option would be for states to nego- 
tiate individual weights for each of the Department 
of Labor's perforraance measures for each 3pA. 

This provides more flexibility but still limits 
the index to the existing measures. This allows 
some trade-off among standards depending upon an 
individual SDA's problem with the standards^ but 
it may not offer complete relief to an SDA who 
wishes to serve a specific target group or use a 
unique intervention strategy. 

3. Add a fixed set of additional measures for all 
SDAs and assign each one a fixed statewide weight. 

4. Vary the additional measures across SDAs but de- 
velop pre-assigned weights for them by fixing 

the combined relative weight of the existing Depart- 
ment of Labor standards; or 

5. Vary the use of both additional measures and weights 
across SDAs. 

TO introduce more flexibility into the negotiation process, 
states can consider one of the three latter alternatives. 
Each of these options offers substantially more latitude for 
negotiation, but they complicate the process and require more 
careful consideration of consistency of application. 
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Again, states are cautioned that these alternative s are a 
riskier approach to adjusting standards . 

In the event that states elect to add measures, they will 
need to determine which additional measure to incorporate into 
the performance index. These additional measures should be 
selected on the basis of their relationship (s) to the particular 
model shortcomings to be addressed or to the goals states 
choose to emphasize, if done on a statewide b£'.;is, this 
determination can be made in conjunction with SDAs each year. 
If done on an individual SDA basis, the state could limit 
the choice of additional measures to a fixed set or allow 
the SDA full/partial flexibility of choice. Statis could 
give the SDA the option of proposing their own measures, m 
either case, SDAs would be required to demonstrate (as dis- 
cussed previously in this Guide) the link between the model 
shortcoming, the problem caused, and the relief provided by 
the newly proposed performance measure. 

Similarly, the establishment of weights on a statewide or SDA 
basis should reflect the relative priority needed for each 
measure in order to address model shortcomings or perceived 
problems. If done statewide, states should consider basing 
their decisions on the collective input of all SDAs. If 
done on an individual SDA basis, each SDA would have to demon- 
strate why the weighted adjustment and its particular value 
were necessary, if this were not possible, each measure would 
receive equal weight. 

It is important to point out that the adoption of a weighted 
performance index does not have to guarantee that additional 
measures or different weights will always be used by each SDA. 
A state could adopt a varying SDA weight and additional measures 
approach and offer it as an opportunity. Unless it could be 
demonstr ated that relief from model-predictid scores was 
warranted and addr essed by the proposed additional standards 
and/or weights, each of the Department of Labor's standards 
would be used with equal weights . There is, in effect, a check 
and balance system directly linked to careful documentation 
of the reason for an adjustment. 

Exhibit H provides an example of a recalculated index using 
different weights for two SDAs for each measure. In the 
example, SDA I has negotiated a higher weight for the welfare 
entered employment rate because of its plan to emphasize 
service to welfare clients. The SDA did not exceed its 
model-derived standard at the end of the year for the welfare 
entered employment measure, but it did improve the average 
wage standard. The SDA did well on the ratio because it did 
very well on the cost measure. (The SDAs actual performance 
was 83.3 percent of the model-derived standard.) 
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EXHIBIT H 

Illustrative Example of an Additive, Weighted Index of Title IIA Adult Program 

Performance for Two SDAs 



SDA I 



(A) (B) (C) 

Assigned Achieve- 

Performance weight of ment 

measure the mea- rate 

sure 



(D) 
Achieve- 
ment rate 
X weight 
of measure 



(B) 
Assigned 
weight of 
the mea- 
sure 



SDA II 



(C) 
Achieve- 
ment 
rate 



(D) 
Achieve- 
ment rate 
X weight 
of measure 



Entered un- 
subsidized 

employment #30 90 

Average wage 
at placement 
standard 

Welfare 
entered 
employment 

rate .50 70 



.20 120 



27.0 



24.0 



35.0 



.20 



.50 



.20 



90 



120 



70 



18.0 



60.0 



14.0 



1 

ui 
1 



Total value 
of index of 
performance = 

Value of in- 
dex of pro- 
gram costs = 

Value of Performance Index 
Value of Index of Costs 



86.0 
83.3 



86.0 



83.3 



1.03 



92.0 
125.0 



92.0 

125.0 
73.6 



Definitions: 

(1) Entered Unsubsidized Employment: The ratio of the number of participants placed in unsubsidized 

jobs upon termination to total number of terminations from the program. 

(2) Average Wage at Placement Standard: Mean beginning hourly wage of unsubsidized job placements 

divided by SDA wage standard times 100. 

(3) Value of Index of Program Costs: Cost of program per unsubsidized entered employment divided 

by average cost per unsubsidized placement established as a standard times 100. 
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SDA II negotiated different weights for the measures with 
average wage at placement being the "most important" measure • 
In this example, the SDA did well on the measure weighted 
most heavily, but it did less well on the overall index be- 
cause its actual cost per entered employment significantly 
exceeded its recalculated model-derived cost standard* Even 
though the SDA achieved the same percentage of the model- 
derived standard for all of the outcome measures, the weights 
in its performance index combined with its relatively poor- 
performance against the cost measure meant that the SDA did 
relatively poorly on the overall index of performance • 



Use of Adjustments to Specific Measures 



Another tool that may be used to make further adjustments beyond 
the Department of Labor's adjustment model is to adjust a 
specific performance measure- One primary example of this 
tool is to adjust the average wage at placement, based on 
variables in the structure and conditions of a local labor 
market • In this case, a scaling technique may be used to 
account for the under- or over-adjusted model-derived wage 
standard* 

The first step is to document variation in the structure of 
labor markets as an area for negotiation- This would involve 
obtaining inter-industry or inter-occupational data to deter- 
mine if the population density adequately captures differences 
in wages paid in urban and rural areas and whether the average 
wage figure used in the adjustment model and wages paid in 
dominant industries were different from those reported in the 
SDA or whether wages paid in dominant occupational areas were 
different from those reported in the SDA. Since within county 
differences are not corrected in this process, the extent 
that such differences can be identified establishes a valid 
area for negotiation. 

The three most viable sources of information on wages paid by 
industry, occupation, or urban and rural area include: 1) the 
1980 Census; 2) area wage surveys conducted by the Bureau of 
Labor Statistics (BLS) and available from ELS regional offices; 
and 3) ES 202 data which contain information on total employ- 
ment and wages paid by industry. Each of these data sources 
is available for public use and are collected through acceptable 
statistical methods. 

Review of Data Sources for Adjusting 

the Average Wage at Placement (See note 5.) 

There are several different sources of wage-related data that can 
potentially be used when attempting to adjust the model-derived 
average wage at placement standard. The major sources that 
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states and SDAs may wish to consider include: 

1. The 1980 Census 

2. The Employment and Wages (ES 202) Program 

3. Area wage surveys conducted by the Bureau of 

Labor Statistics 

4. The Job Bank data maintained by the Employment 

Service 

5. Locally iiiitiated surveys 



The 1980 Census . The Census can be a very useful source of in- 
formation on local area employment, earnings, and wages. 
Every ten years, the Census Bureau collects detailed informa- 
tion on population, housing, and employment related character- 
istics. These data are then made available for public use 
in several different forms, differentiated primarily by 
either subject matter or size. For the sake of analytic 
efficiency, a 1 in 100 and a 1 in 1,000 sample tape is made 
available. These tapes contain detailed information on 
individual employment, earnings, and wages experiences that can 
be used to generate local area estimates of such measures as 
average wages, industry composition, and occupational employ- 
ment. As such, these data represent a viable source of infor- 
mation for states and SDAs interested in further adjusting the 
average wage at placement standard. 

There are, however, certain practical shortcomings with taking 
this approach. The first is that acquisition of the data for 
selected analyses of area wages can be difficult unless the 
state or SDA has the computer facilities needed to process 
the data. Although hard-copy publications of Census data are 
available, they typically do not include detailed data on a 
local basis consistent with the boundaries of SDAs. In 
addition, while each state maintains a Census Data Center, the 
types of services provided by them vary across states and may 
not include those needed to obtain the requisite information. 

The Employment and Wages (ES 202) Program . The Employment and 
Wages Program, commonly referred to as the ES 202 Program, is 
one of several federal-state cooperative programs. Using 
quarterly reports submitted by state employment agencies, the 
Bureau of Labor Statistics summarizes employment and wage 
data for workers covered by the unemployment insurance laws 
and for civilian workers covered by the program of Unemployment 
Compensation for Federal Employees. This program and its 
data set provide a comprehensive and accurate source of employ- 
ment and wage data, by industry and at the national, state, and 
local levels. 

Since each state maintains and processes its own 202 data, it 
is possible for states and SDAs to secure the information for 
performance standards purposes. There are, however, certain 
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weaknesses of this approach. First, the level of detailed 
data varies and will have to be checked on a state-by-state 
basis. Second, self-employed persons in retail trade, con- 
struction, and service industries are not included, although 
this should pose little difficulty for performance standards 
purposes. Third and last, data processing capabilities vary 
by state and will have to be carefully checked prior to 
attempting to obtain the data. 

Area Wage Surveys . Area wage surveys are conducted by the Bureau 
of Labor Statistics, usually for selected occupations in 
selected areas of the country. On an occupational basis, 
these data are very accurate and can be useful for further 
adjusting average wage standards to account for occupational 
differences. However, these data are not typically available 
on an SDA basis and do not cover all areas of the country. 

Job Bank Data . The Employment Service maintains records of 
all jobs listed with it, including the occupation and wage 
rate. Although these data can be of potential use in adjusting 
the average wage perfoinnance standard, their chief short- 
coming stems from their lack of representativeness. Since 
only a small proportion of all job openings are listed with 
the Employment Service, these data cannot be relied upon to 
generate accurate estimates of occupational wages. 

State and SDA Management Information Systems . State and SDA 
management information systems retrieve, among other data 
items, information on the placement wages of former partici- 
pants. As such, they represent one potential source of data 
for adjusting the average wage at placement. Like Job Bank 
data, however, these data also present problems of represen- 
tativeness since they are confined to the placement outcomes of 
JTPA participants and not all resident workers within an SDA. 

Locally Initiated Surveys . The last potential source of wage 
data is locally initiated surveys. In this case, the state 
and/or SDA would initiate a survey of local employers to de- 
termine the average wages paid to workers. While this approach 
can yield important and accurate wage data, it will have to 
be conducted within the constraints imposed by the statistical 
sampling and data collection requirements. 

Regardless of the source, it is necessary to obtain the wage 
data necessary to measure the difference between the average area 
wage and that considered more appropriate for the SDA. If, 
for example, it were being argued that differences between 
rural and urban areas were not adequately represented in the 
adjustment model, then wage data for both geographic units 
would have to be obtained. The same case would prevail for 
industry and/or occupational wage structure. 
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Once the data were obtained, two similar, but somewhat dif- 
ferent scaling techniques could be used* The first would be 
to divide the average wage for the unaccounted for labor mar- 
ket structure (e.g., rural area) ^ by the average wage for the 
SDA used in the adjustment model. The result would be a 
fraction that could be multiplied by the model-derived wage 
standard to derive a new scaled version of the same standard. 
In essence, the model-derived wage standard would be scaled by 
the extent to which the wage associated with the unaccounted 
for labor market structure was greater or less than the average 
area wage. 

As an example y consider an SDA located in a rural area of a 
state that has demonstrated that its wages are considerably 
below those in the area used to calculate the wage level used 
in the model. (Remember ^ population density is in the model 
and represents rural/urban differences.) Further assume that 
the rural average wage is $3.50, that the area wage used in 
the model was $4.50, and that the model-derived performance 
score was $4.25. In this case, the ratio of the rural to area 
wage would be: $3.50/$4.50 = 77 percent. By multiplying 77 
percent by the model-derived performance score of $4.25, the 
scaled wage standard would be set at $3.27. The same pro- 
cedure would hold for any other adjustment made as a result 
of labor market structure. 

The chief disadvantage of this approach is that it does not 
account for the SDA's wage position relative to that of the 
country. Since the adjustment model fits SDA performance 
behavior to a national model, deviations from national 
relationships may be more appropriate to use. Thus, rather than 
relying on a SDA's deviation from the area average, the devia- 
tion expressed above as a ratio would be scaled in the same 
manner that the service to youth standard is calculated. 
Thus, the 77 percent discussed above would be divided by the 
ratio of average wages in all rural areas in the U.S. to the 
average wage in the U.S. 

As an example, assume that the average wage in rural areas of 
the U.S. is $3.75 and that the U.S. average waae is $5.00. 
Dividing these two results in a ratio of 75 percent. If this 
75 percent is divided into the 77 percent from above, the 
adjustment would be 1.02. Thus, because there was greater 
proximity between rural to area wages in the SDA than there 
was in the U.S., the SDA's wage standard would be adjusted 
upwards. 

This adjustment to the average wage at placement is only one 
example of an adjustment to a specific measure. States and 
SDAs may identify other measures which require hand-tailoring 
to fit local circumstances. In some cases, states and SDAs 
may want to negotiate "trade-offs" of a higher standard on 
one measure for a lower standard on another measure. 
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Increasing the Tolerance Intervals 



As discussed earlier, the tolerance intervals of each of the 
performance measures represent the degree of imprecision in 
the models. To some extent, this imprecision is due to 
those factors that affect performance but are unaccounted for 
in the model. 

The process used to increase the tolerance intervals is very 
arbitrary. The same increase in the tolerance intervals 
would be used statewide for all SDAs. One way to increase the 
tolerance range would be based on past experience within 
the state. The state could calculate the Standard Deviation 
in performance for the prior year for the measure in question 
among all SDAs and use that (or a portion of that deviation) 
as the increased tolerance interval for the specific measure. 

This method is, however, among the least desirable since there 
is not a quantifiable method to determine how much the inter- 
vals should be changed. As a result, changes to the model's 
tolerance intervals are best used when no other means of 
adjustment are available. This might be the case for extreme 
values and unexpected events that cannot be readily linked 
to a specific model shortcoming. Increasing tolerance limits 
might be an appropriate fix for extreme values because rela- 
tivity increases as predicted values are further from the mean. 



Negotiation and Six Percent Incentive Funds 



Under Title II-A, six percent of the funds are available as an 
incentive for achieving good performance and to provide 
technical assistance to those areas not receiving incentive 
funds. In several instances, states have elected to use a 
portion of their 6 percent funds to encourage services to the 
"hard-to-serve" target groups that would not otherwise be 
served. This is intended, in part, to address any disin- 
centives that may be introduced by the current performance 
standards, including the model-derived scores. Although the 
use of incentive funds may increase services to selected target 
groups, it will not provide relief from what may be limi- 
tations introduced by the model-derived performance scores. 
The reason for this is that while an SDA may receive incentive 
funds for achieving an additional standard (such as an input 
standard) , it still must achieve its other performance stan- 
dards. Thus, incentive funds alone should not be expected to 
address the limitations caused by model shortcomings. Rather, 
incentive funds should be planned in conjunction with the 
development of negotiation strategies to form a consistent 
approach to performance standards achievement. 
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The necessity of jointly planning both activities is also 
evident from the negotiation tools presented earlier. For 
this reason, there needs to be a high degree of consistency 
between each. Take for example the addition of measures as 
a negotiation tool. If the same additional measure (s) is 
(are) used for incentive funds, it may be necessary to set 
two different standards, or a range, the lower one for meeting 
performance and the higher to be eligible for incentive awards. 



ESTABLISHING THE FRAMEWORK FOR NEGOTIATION ; 
AN OVERVIEW 



Basic Elements of a Negotiation Framework 



The establishment of performance standards is critically linked 
to an understanding of the employability development process 
and the conditions under which it takes place. Factors that 
describe this process have been incorporated into a series 
of statistical adjustment models designed to neutralize their 
effects on an SDA's potential performance capability. These 
factors may not capture the full range of variables that affect 
the employment and earnings of program participants. Nego- 
tiation should identify these additional factors and cast them 
in an employability context so that the performance potential 
of any SDA can be more fully described. This requires an 
understanding of the problems eligible participants face in 
the labor market and how they relate to both program design 
and performance outcomes . 

It is also important to stress that negotiation must be an 

open, public process. The need for negotiation must be articulated 

publicly and understood by all parties involved. 

Sound planning and management infoinnation systems are a 
critical ingredient for inf Ginned negotiating. Both systems 
assist in identifying any shortcomings of the statistical 
adjustment models and developing revisions to both model- 
derived performance scores and program targeting and design. 

Exhibit E presented the necessary key steps to demonstrate 
the program relevance of each model limitation and the need 
for negotiation. These tasks largely embody the program plan- 
ning process and should not represent an extra effort carried 
out just for performance planning. Rather, they should be 
part of the on-going JTPA planning process that can be drawn 
upon as needed. For negotiation to proceed, particular program- 
matic problems and potential solutions must be linked. 
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Sound planning needs supporting data. In addition to labor 
market data that may be obtained from institutional sources 
and from special surveys, important information is contained 
in the management information system. This system is the 
repository of information on local and state program experiences 
to date. It represents a potentially rich source of knowledge 
from which the data needed to support negotiation and plan 
future programs can be retrieved. 

For example, several states and SDAs have raised concern over 
the exclusion of selected population sub-groups from the 
statistical adjustment models that remain important in ex- 
plaining performance. As has been discussed, this program- 
matic problem can become the basis for negotiation only after 
it is demonstrated to be causally linked to a program's ability 
to perform satisfactorily. One way to demonstrate this is 
to rely upon previous years' program data to identify the 
extent to which these individuals have been served, in what 
programs, and at what level of success. 

If, for example, negotiation focused upon unmeasured character- 
istics such as functional literacy, then this information will 
have to be collected and used both to monitor performance and 
re-calculate performance at the end of the program year. 
Thus, the management information system must be structured to 
capture the key components of the employability development 
process. In most instances, data beyond that required for 
national reporting must be routinely collected. 

The management information system serves as the basis for 
monitoring on-going performance. Once a negotiated perfor- 
mance level is agreed upon, data on the areas upon which per- 
formance was based must be retrieved. 

Exhibit I presents a schematic diagram of the steps for 
establishing a unified negotiation framework that adheres to 
the principles outlined thus far. These steps would form a 
state's negotiation policy and an SDA's plan to document the 
need for negotiation. 

The first step will be to determine the areas that will be 
considered for negotiation. These may include, but are not 
necessarily limited to: (1) unmeasured characteristics; and/or 
excluded population sub-groups; (2) program intensity or 
"quality" (duratio.i, costs, mix); (3) youth positive termina- 
tion rate; (4) average wage at placement; (5) extreme values; 
and (6) unexpected events. 

The second step is to specify how the problems that may be 
caused by the standards (e.g., limited targeting flexibility) 
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EXHIBIT I 



Key Steps for Establishing a Unifi 

II 



IDENTIFY POTENTIAL AREAS 
FOR NEGOTIATION 



1. 



2. 



3. 



5, 
6. 



Unmeasured character- 
istics and/or ex- 
cluded population 
sub-groups 

Program intensity or 
" qual i ty " (dura- 
tion, costs, mix) 

Youth positive termi- 
nation rate 

Average wage at 
placement 

Extreme values 

Unexpected events 



DETERMINE REQUIREMENTS 
FOR ESTABLISHING LINK- 
AGE BETWEEN MODEL LIMI- 
TATIONS AND THEIR POTEN- 
TIAL EFFECTS 



POTENTIAL EFFECTS 



2. 



3. 



Limited targeting 

flexibility 
Limited program in- 
tensity or con- 
strained program 
mix 

Limited connection 
between standards 
and positive ter- 
mination rate 

4. No adjustments to 

specific measures 

5. No adjustment for 

extreme values 

6. No adjustment for 

unexgect3d_events^ 



REQUIREMENTS 



1. Identify nature of 

problem and link 
to model short- 
coming 

2. Determine alterna- 

tive targeting and 
program design 

3. Determine implica- 

tions for perfor- 
mance 



79 



Negotiation Framework 



III IV 
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will be identified, including their relationship to particular 
model limitations and their implications for program targeting, 
design, and performance. The third step necessary for 
establishing a negotiation framework is to select a tool for 
further adjusting model-predicted performance standards. 
Available options include adding new standards, weighted 
averages, weighted indices, adjustments to specific measures, 
or increasing the model-derived tolerance intervals. The last 
step is to determine the data collection and reporting re- 
quirements necessary to support continued performance stan- 
dards negotiation. 

Once these basic planning steps have been taken, states and 
SDAs will be in a position to negotiate performance on a regular 
basis with the support of both program planning systems and 
management information systems. 



Documentation and Information Sources 



Key to the process of performance standards negotiation is 
sound documentation. SDAs must be able to demonstrate: 

1) that a problem exists; 

2) how it affects their model-predicted performance, 

and; 

3) to what extent adjustments are necessary. 

This documentation requires careful planning and the use of 
reliable da^a to support it. 

Several reliable data sources ar*:; available to support the 
negotiation process. The most important and promising source 
IS the management information system (MIS) , operated either 
as part of a statewide system or independently by SDAs. This 
MIS is the repository of current and historical data on the 
characteristics of participants served, the programs in which 
they were enrolled, and rhe types of outcomes they experienced. 
As such, it can prove a valuabl'.j source for documenting 
constraints encountered in serving certain types of par- 
ticipants and the need for new and/or more extensive' designs. 

In many cases, however, the existing MIS may not contain the 
participant characteristic data needed to demonstrate the 
unique problems faced by selected sub-groups of the population 
(e.g., teenage mothers, those with limited functional 
literacy), in these instances, two strategies may be considered. 

The first strategy is to modify the MIS to capture the full 
range of additional data needed to develop an employ ability 



81 



-66- 



development process for participants. These data would in- 
clude participant characteristics not included in the adjust- 
ment model (i.e.r teenage mothers or displaced homemakers) . 
They may also include human capital attributes such as more 
detailed educational and age data (which serve as a proxy 
for experience) that may be already routinely collected. 
Other measures of employability such as basic skill levels ^ 
health/disability status r competency levels in v*orld of work 
knowledge r work maturity ^ and vocational skills are not 
typically collected but may also be appropriately added to 
the MIS and used for documentation. 

Complete specification of the full range of additional data 
for the MIS can best be accomplished by: (1) developing a 
composite list^ (2) assessing each variable in terms of its 
relationship to employment and earnings and feasibility of 
collection; and (3) determining those that are most impor- 
tant and practical to obtain. 

There are two major advantages of expanding the MIS. The 
first is that it will provide the basis for monitoring program 
performance both during and at the end of the year. This will 
be important in determining the extent of the adjustments to 
the performance standards. The second advantage is that 
initial expansion of the MIS will provide the basis for docu- 
menting the need for negotiation in future years. 

The second strategy for obtaining data when the existing MIS 
is insufficient is to consult sources of information that will 
likely contain the requisite data. These include the following 

1. Local school systems which have access to student 

characteristics y performance^ and dropout data; 

2. Local labor market data^ with particular reference 

to the problems and barriers faced by population 
sub-groups; 

3. The evaluation and program analysis literature ^ 

which can provide evidence regarding the effective- 
ness of alternative program strategies for the 
economically disadvantaged; and 

4. Data from comparable SDAs^ which may be relevant 

to the local situation. 

The information used and conclusions reached will have to be 
credible and defensible r as specified in the Department of 
Labor parameters discussed in Chapter 1. 
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STATES AND SDAs; ROLES AND RESPONSIBILITIES 

To this point r the major elements of a negotiation process 
have been presented and discussed. The end result of this 
negotiation process will depend on how states and SDAs ex- 
ercise their respective roles and responsibilities under 
JTPA. 



Stat;^ Roles and Responsibilities 

Under JTPA^ states are given responsibility for setting per- 
formance standards and establishing planning ^ data collection^ 
and reporting requirements. In view of this^ the natural 
starting point for negotiation is the state. 

If states choose to incorporate negotiation into their per- 
formance standards-setting process ^ they will have to initiate 
a decision-making framework for resolving several important 
issues, including: (1) SDA-PIC involvement in decisions about 
the nature and structure of negotiation; (2) selecting a 
negotiation tool; (3) documentation requirements both in the 
job training plan and at the end of the year; (4) data col- 
lection and reporting requirements; and (5) an arbitration 
process for making final determinations over negotiated per- 
formance* Each of these issues is discussed below. 



SDA-PIC Involvement 

In designing a negotiation process , states will first have to 
deteirmine how they will involve SDAs/PICs in each of the major 
issue areas. The options include: 

1) States may simply determine the policies and pro- 
cedures themselves, without negotiating with SDAs. 

2) States may choose to develop a "bottom- up" process 
whereby each of the major decision areas would be 
jointly reviewed by both state and SDA staff and 
then resolved. The advantage of this approach is 
that it will build support for the negotiation 
process and establish a uniform appreciation and 
understanding of its importance and mechanics. 

3) States will need to decide whether they wish to treat 
all SDAs uniformly throughout the State, or whether 
thay wish to deal with SDAs individually. 
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4) Another option for states is to establish a request 
and approval process, SDAs request changes ^ either 
through the job training plan or as part of the end 
of the year recalculation process ^ and states 
determine whether or not these changes will be 
accepted • 

Several possible forums exist for states to involve SDAs in 
this process. One is the State Job Training Coordinating 
Council (SJTCC) and its committees which have SDA represen- 
tation. Another is an SDA workgroup or directors group that 
the state may regularly operate as part of its management of 
JTPA. In the event that either of these is not possible or 
appropriate y states may wish to consider establishing a specific 
SDA workgroup for addressing performance standards negotiation. 



Selecting a Negotiation Tool 



Once states decide that negotiation is appropriate ^ they will 
have to deteirmine the specific adjustment strategy or stra- 
tegies that they will use to modify the model-derived per- 
formance scores. Because the adjustment selected will 
influence the types of documentation requirements ^ data col- 
lectionr and reporting that will be necessary ^ it should be 
decided early in the planning process. 

Five basic adjustment tools have been presented for con- 
sider a'*'.ion, (Other tools may certainly be used^ if states so 
choose.) The five options presented here include: 1) the 
addition of performance measures; 2) the use of weighted 
averages of performance; 3) the use of weighted indices; 
4) adjustments to specific measures (i.e., average wage at 
placement) ; and 5) increasing the tolerance levels established 
by the adjustment model. Exhibit J presents the major advan- 
tages of each of these alternatives. 

It is important to note that there is no one correct method 
for states to choose. Given the strengths and weaknesses of 
each alternative f the one most appropriate for specific state 
use should be selected. States may wish to allow SDAs to 
choose among the negotiation tools or decide to use any one 
(or more) of the tools statewide for all SDAs. 
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EXHIBI * J 



Advantages and Disadvantages of Alternative Adjustment Tools 



Adjustment Tool 

1. Additional measures 



Advantages 

1. Simple to design 

2. Ease of Implementation 

3. Allow for targeting 

standards on particu- 
lar groups 

4. Useful for statewide 

application 



Disadvantages 

1. Does not provide relief 
from potentially adverse 
effects of model short- 
comings 



2. Weighted averages of 1. limits negotiation to only 1. Focuses on a single measure 

performance those standards under con- separately 

sideration 

2. Can address both target 

group and program in- 
tensity (mix) adjustments 

3. Useful for individual SDA 

application 



3. Weighted indices 



1. Focuses on composite per- 

formance picture 

2. Offers Maximum flexibility 

to states and SDAs 

3. Useful for statewide appli- 

cation 



1. Most complex to design and 

implement 

2. May promote inconsistencies 

or inequitable treatment of 
some SDAs versus others 

3. May make states vulnerable 

to challenges 



4. Adjustments to specific 
measure s (i.e., aver age 
wage at placement) 



1. Useful for individual SDA 

application 

2. Accounts for local dif- 

ferences (in labor market 
conditions, etc. ) 



1. May create inconsistencies 
among states and/ or SDAs 



5. Increase tolerance levels 



1. Simple to implement 

2. Useful for individual SDA 

application 



1. Lacks objectivity 
2 • Non-quantitative 
3. May create difficulties in 
resolution of negotiation 



Documentation Requirements 



Although a state may decide to incorporate negotiation into 
?ts perfoimance standards-setting process r there is no reason 
to expect that all SDAs will have their model-predicted per- 
formance adjusted. Such adjustments will only be warranted 
if they can document how the problem relates to performance 
outcomes. The overall requirements necessary to substantiate 
the need for negotiation consist of resolving three 
issues. The first is a specification of the areas that 
states will consider for negotiation. 

The second issue is the development of policies to determine 
whether negotiation within the various areas is warranted. 
States will need first to specify those documentation require- 
ments that are essential to negotiation. States may wish to 
consider the matrix outlined in Exhibit E as the foundation 
for their planning guidelines or may design other similar 
ones. It is important to emphasize that effective documen- 
tation will require establishing systemmatic links between 
the particular problems SDAs believe they have^ model-derived 
performance y and proposed changes to both program client groups 
and performance. 

The third issue is the delineation of appropriate data sources 
for use by SDAs. States w-Jl need to be assured that the data 
used by SDAs are of sufficient quality to support the need for 
negotiation. As discussed earlier ^ one reliable source of data 
is the management information system which reflects actual pro- 
gram experience statewide. 

In several instances, however r the MIS may not contain the data 
SDAs may need to substantiate negotiation. In these cases # 
SDAs may choose to use data and/or findings from local school 
systems, from local labor market information sources, or from 
the evaluation literature. 

Data sources are also important for quantifying adjustments to 
the model-predicted performance standards. In the cases of 
additional perfoinnance measures and weighted averages, 
estimates of either new performance measure standards or ad- 
justed current standards must be made. As above, to ensure 
that the proposed SDA performance standards are credible, 
states should require guidelines governing the types of 
data that can be used. 

Here too, the MIS, local labor market information, local pro- 
gram or school system data, or the evaluation literature can 
be used. The manner in which they will be used is governed by 
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the particular adjustment tool and should also adhere to the 
parameters for judging the acceptability of the data source 
itself. 



Data Collection and Reporting Requirements 



The establishment of performance standards is the beginning of 
an annual ongoing process of program reporting and monitoring. 
Just as the perfoinnance standards are monitored during the pro- 
gram year^ any adjustments to them as well as the basis for 
these adjustments^ also need to be monitored. Since the basis 
for negotiation will often depend upon data not yet included 
in the MIS^ states will have to consider requiring the col- 
lection of such data on the local level and its reporting on 
a regular basis. The addition of new data collection require- 
ments should take place early in the development of a nego- 
tiation policy. Once states have determined the areas they 
will consider for negotiation and the particular negotiation 
tool(s) they will use^ the implications for data collection 
can be addressed. 

TO a large extent^ these additional data requirements can 
be incorporated into the existing MIS insofar as participant 
characteristics are concerned. To the extent that unexpected 
events or unique features of local labor markets are included^ 
SLAs would be required to document them at the end of the year. 

The addition of new data collection requirements will allow 
states (and SDAs) to monitor program and performance activity 
in the particular areas which were negotiated. If ^ for example^ 
performance standards were negotiated to account for dispro- 
portionately large shares of services to those with limited 
functional literacy^ then such information would be gathered 
through the assessment process (i.e.. General Aptitude Test 
Battery (GATE), etc.) added to the MIS and monitored on a 
continuous basis. This will allow states to determine if SDA 
plans were implemented as planned , and re-calculate perfoinnance 
at the end of the program year on the basis of ajtual data. 



Establishing an Arbitration Process 



Once the basic elements of a negotiation process have been 
determined y states must determine the manner in which requests 
for adjustments will be made and resolved. This will consist 
of a set of policies governing when such requests must be sub- 
mitted, to whom the requests will be forwarded, how decisions 
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will be made^ and whatr if any^ appeal process will be followed. 
It is likely that requests for model adjustments would be sent 
to the same individuals or groups currently responsible for 
setting SDA performance standards. This may consist of one 
person responsible for an SDA or a committee that regularly 
meets to review performance and make corrective action 
recommendations . 

It is possible that state staff may decide not to approve a 
request for performance standards adjustments. In these 
instances r the state will have to decide whether SDAs will have 
an opportunity for appeal. If the state determines that appeals 
are desirable ^ then it must identify the requirements for such 
appeals r the individual to whom appeals would be made^ and how 
appeals would be processed. 

At a minimum^ the appeal process requirements should include 
a clear explanation from the SDA as to why it believed that 
adjustments were necessary and why state staff did not permit 
the adjustments. 



SDA Roles and Responsibilities 



To a large extent ^ the responsibilities of SDAs will be de- 
fined by states negotiation policies. In some instances ^ SDAs 
and States will make these decisions jointly. The areas for 
negotiation r the negotiation tools ^ reporting and data col- 
lection requirements y and the arbitration process will most 
likely be decided early in the planning process. If SDAs wish 
to play a part in these decisions ^ they should so inform the 
states. If the state has made these decisions unilaterally , 
it will be the responsibility of SDAs to adhere to these 
requirements in developing a request for adjustments to the 
model-derived performance scores. 

In some instances ^ states may decide not to develop formal 
negotiation policies and SDAs will have to take the initiative. 
In these instances ^ the same decisions regarding documentation 
and reporting and data collection will have to be made by the 
SDA and then followed in the development of an adjustment 
request. 

One issue that is unique to SDAs is the involvement of PICs. 
A;5 is the case with other planning and policy responsibilities ^ 
any request for adjustments to the model-derived performance 
scores must be approved first by the local PIC. 
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SDAs and PICs must assume the responsibility for documenting 
the need for an adjustment and determining how much adjust- 
ment may be warranted. There are other responsibilities that 
SDAs and PICs must consider: collecting data on a proposed 
adjustment is important. In addition, the PIC and SDA must 
monitor performance and ensure that either the type of service 
and/or the target group documented as being underserved is 
actually served. 

States will not grant an adjustment negotiated during the 
planning stage if the group targeted or service planned does 
not materialize. 
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REFERENCE NOTES 



1. See, for example: 

Comptroller General, Labor Market Problems of Teenagers 

Result Largely from Doing Poorly in School , General 
Accounting Office, Washington, D.C., March 1982. 

National Commission on Excellence in Education, A Nation at 
Risk: The Imperative for Educational Reform , 
April 1983. 

Sum, Andrew, Educational Attainment, Academic Ability, and the 

Employment and Earnings of Young Persons in the United 
States: Recent Trends and Findings of Recent Research , 
Center for Labor Market Studies, Northeastern 
University, Boston, Ma., 1983. 

Taggart, Robert, Youth Jobs Programs: The Critical Need for 
a Comprehensive Strategy , Remediation and Training 
Institute, Alexandria, Va., 1983. 

2. It is important to point out that the Department 
of Labor's selection of variables to include in the adjustment 
models has been based on a systematic, comprehensive assess- 
ment, including both statistical tests and the advice of state 
and SDA practitioners. In many cases, variables were excluded 
because they did not exhibit a statistically significant 
relationship to the performance measures. In other cases, 
such as teenage mothers, too few participants were served to 
warrant data collection and thus were excluded from the model. 
In still other instances, such as functional literacy, the 
collection of data was not deemed practical from a national 
reporting perspective. 

3. For PY86, the Performance Standards Advisory 
Committee has recommended to the Department of Labor that 
attainment of youth competencies be reported separately in the 
Job Training Partnership Act Annual Status Report (JASR) . 
However, it can not be incorporated in any model until PY87. 

4. At its June 1985 meeting, the Performance Stan- 
dards Advisory Committee recommended to the Deparrjuent of 
Labor that post-program performance measures be uised for Title 
II-A and Title III programs. Data collection would begin in 
PY86 and standards would be set initially in PY88. 
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5. Also, two good sources for reviewing in detail, 
the availability of labor market information and the conduct 
of local employer surveys include: 

• "Issues Related to the Development and Use of Employer 
Surveys At the Local Level," edited by Paul E. 
Harrington, et al* Proceedings of the Employer 
Survey Conferences, February 13, 1980 and April 10 , 
1980 . Center for Labor Market Studies, Northeastern 
University and Massachusetts Occupe.tional Informa- 
tion Coordinating Committee, August 1981. 

• "Labor Market and Occupational Information for CETA 
Prime Sponsor Poll jymaking. Planning, and Program 
Operations," Paul F Harrington and Andrew M. Sum, 
Prepared for the National Occupational Information 
Coordinating Committee, Washington, D.C., by the Center 
for Labor Market Studies, Northeastern University, 
Boston, Ma., 1980. 
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96 STAT. 1334 PUBLIC LAW <)7-:}00~OCT. 13, 1982 



PntrORMANCB BTANVAMDS 

Ssa 106. (a) The Congrett recognizee that job training ie an ^ use isie. 
investment ' in human capital and not an expense. In order to 
ddtermine whether that investment has been productive, the Con* 
grass finds that-* 

(1) it is fi^ntial that criteria for measuring the return on this 
in vestment be developed; and 

(2) the basic rstuni on the investment is to be measured by 
the increased amplogrment and earnings of participants and the 
reductions in vrellare dependency. 

(bXl) The basic measure of p«rfonnance for adult training pro* AduUtninioc 
gsama under title Did the incTMM in <ttipkm prognM. 
the i^uctioiia in vieUm dependenqr reeulting firm 
the program; la cMer to determine whether theee nasie measurss 
are adueved, the Skrstavy shaU prescribe sti^^ 
a9propriate*fiictOK» which nMqr^indude (A) p lac ement in unsubsi- 
dim emplqfoient» (B) lUntwn in unsubddised empkmnent* (Q 
the luueass in eamino^ ihrhidina houriv wages, and (D) reduction 
in the number of individuals and iSuniUss receiving cash weliars 
payments and the amounts of audi pqrments. 



Youth profnm*. (2) In prescribing standards under this section the Secretary shall 
also designate factors for evaluating the performance of youth 
programs which, in addition to appropriate utilization of the factors 
described in paragraph (1), shall be (A) attainment of recognized 
employment competencies recognized by the private industry coun* 
cil, (B) elementary, secondary, and postsecondary school completion, 
or the equivalent thereof, and (C) enrollment in other training 
programs or apprenticeships, or enlistment in the Armed Forces. 

(3) The standards shall include provisions governing— 

(A) the base period prior to program participation that will be 
used; 

(B) a represenUtive period after termination from the pro- 
gram that is a reasonable indicator of postprogram earnings 
and cash welfare payment reductions; and 

(O cost^fTective methods for obtaining such daU as is neces- 
sary to carrv out this section, which, notwithstanding any other 
provision of law, may include access to earnings records, State 
employment security records. Federal Insurance Contributions 
Act records. State aid to families with dependent children rec* 
Olds, statistiod sampling techniques, and similar records or 
measures* 

(4) The Secretary shall prescribe performance sUndards relating 
gross program expenditures to various performance measures. 

(c) Within six months after the date of the enactment of this Act, 
the Secretary shall establish initial performance sUndards which 
are designed to contribute to the achievement of the performance 
goals set forth in subsection (bXl), based upon daU accumulated 
Am/, p. \x>r under the Comprehensive Employment and Training Act, from the 
National Commission for Employment Policy, and from other appro- 
priate sources. In the development of the initial standards under 
this subsection, the Secretary shall relate ktoss program expendi- 
tures to the accomplishment of program goals set forth in subsection 
(bXl). 
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PUBLIC LAW <>7-3()0-OCT. 18, 1982 



96 STAT. 1335 



K<*purt to 
CunxreM. 



MudiHcution. 



Ki*pi>rt to 



(dXl) The Secretary shall, not Sater than January 31, 1984. pre- 
scribe performance standards for the first program year under this 
Act to measure the results of the participation in the program to 
achieve the goals set forth in su^/8ection (bXl) based upon the initial 
standards established in subsection (c). 

(2) The Secretary, not later than six months after the completion 
of the first two program years, shall prepare and submit a report to 
the Congress containing the performance standards established 
under paragraph (1) of this subsection, together with an analysis of 
the manner in which the performance standards contribute to the 
achievement of the goals set forth in subsection (bXl), including the 
relative importance of each standard to the accomplishment of such 
goals. 

(3) The Secretary shall prescribe variations in performance stand- 
ards for special populations to be served, including Native Ameri* 
cans, migrant and seasonal farmworkers, and ex^offenders, taking 
into account their special circumstances. 

(4XA) The Secretary may modify the performance standards under 
this subsection not more often than once every two program years 
and such fhodifications shall not be retroactive. 

(B) The Secretary shall prepore and submit a report to the Con- 
gress containing any modifications established under subparagraph 
(A), and the reasons for such modifications. 



V«ri«tion8 of 
ttundards 



Development of 
standards. 



(e) Each Governor may prescribe, within parameters established 
hy the Secretary, variations in the standards under this subsection 
based upon specific economic, geographic, and demographic factors 
in the State and in service delivery areas within the State, the 
characteristics of the population to be served, and the type of 
services to be |>rovided« 

(0 The National Commission for Employment Policy shall (1) 
advise the Secretary in the development of performance standards 
under this section for measuring results of participation in job 
training and in the, development of parameters for variations of 
such standards referred to in subsection (e), (2) evaluate the useful- 
ness of such standards as measurss of desired performance, and (3) 
evaluate the impacts of such standards (intenocd or otherwise) on 
the dMioe of who is served* what services are provided, and the cost 
of such servioss in isrvioe delivery areas. 

(g) The Sscrstaryshall prescribe performance standards for pro- 
grama under title ul based on placement and retention in unsubsi- Atte p. I364. 
diasdemployiDMt 

(hXl) Tm Governor shall provide technical assista nce to programs 
which do not meet performance criteria. If the failure to meet 
l^ornanes standards persists for a second year, the Governor 
shall impose a rsorganisation plan. Such plan inay restructure the 
private indostty council, prohibit the use of designated service 
providers or make such other changes as the Governor deems 
necessary to improve performance. The Governor may also select an 
alternate entity to ^minister the prsgram for the service delivery 
area. 

(2) The alternate administrative entity may be a newly formed 
pnvate^uidustrv council or any agenof jointly selected by the Gover- 
nor and the duef elected official of the largest unit of general local 
(ovemment in the eenrice delivery ersa. 

(3) No change may be made under this subsection without an 
opportimity for a hearingbefors a hearing officer. 

(4) pie decision of the Ck>vernor may be appealed to the Secretary Appaai 
who shaU make a final decision witmn 60 days of the receipt of the 
appeal. 



Raorfmnixaiton 
plan. 



ERIC 
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